[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:08 dist-test-slave-1lvv krb5kdc[3105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:08 dist-test-slave-1lvv krb5kdc[3105](info): set up 2 sockets
Sep 25 05:10:08 dist-test-slave-1lvv krb5kdc[3105](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 05:10:09.439738 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:09 dist-test-slave-1lvv krb5kdc[3105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:09.569540 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:35041
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:35041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:09.655839 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:09.656080 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:09.656121 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:09.657363 3118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:09.657438 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:09.657465 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:09.657497 3118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:09.657527 3118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:09.659092 3118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:35041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:35041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:09.659385 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:09.659734 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:09.665896 3126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:09.666723 3124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:09.666847 3123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:09.667490 3118 server_base.cc:1034] running on GCE node
I20260925 05:10:09.667826 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:09.668169 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:09.669338 3118 hybrid_clock.cc:648] HybridClock initialized: now 1790313009669303 us; error 153 us; skew 500 ppm
Sep 25 05:10:09 dist-test-slave-1lvv krb5kdc[3105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:09.672057 3118 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:09.672518 3118 webserver.cc:460] Webserver started at http://127.3.6.190:36709/ using document root <none> and password file <none>
I20260925 05:10:09.672730 3118 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:09.672770 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:09.672863 3118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:09.674026 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "98665c3e030c44768014c0c1872c315a"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-1lvv"
server_key: "a630102a3bc783592988e6e9cad4cef2"
server_key_iv: "ea948c9b7ccf3471efee73a6b5a1ab0a"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.674654 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "98665c3e030c44768014c0c1872c315a"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-1lvv"
server_key: "a630102a3bc783592988e6e9cad4cef2"
server_key_iv: "ea948c9b7ccf3471efee73a6b5a1ab0a"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.676124 3118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:09.676844 3133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:09.677030 3118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:09.677119 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "98665c3e030c44768014c0c1872c315a"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-1lvv"
server_key: "a630102a3bc783592988e6e9cad4cef2"
server_key_iv: "ea948c9b7ccf3471efee73a6b5a1ab0a"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.677222 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:09.693315 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:09.695587 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:09.695887 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:09.699961 3118 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:35041
I20260925 05:10:09.700028 3185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:35041 every 8 connection(s)
I20260925 05:10:09.700342 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:09.701201 3186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:09.703922 3186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a: Bootstrap starting.
I20260925 05:10:09.704649 3186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:09.704978 3186 log.cc:824] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:09.705832 3186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a: No bootstrap required, opened a new log
I20260925 05:10:09.706712 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3118
I20260925 05:10:09.706863 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:09.707794 3098 external_mini_cluster.cc:1428] Setting key 8c1a3a0011eda97303a2ccc3e0fee4d8
I20260925 05:10:09.708477 3186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98665c3e030c44768014c0c1872c315a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35041 } }
I20260925 05:10:09.708751 3186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:09.708801 3186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98665c3e030c44768014c0c1872c315a, State: Initialized, Role: FOLLOWER
I20260925 05:10:09.708968 3186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98665c3e030c44768014c0c1872c315a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35041 } }
I20260925 05:10:09.709096 3186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:09.709147 3186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:09.709208 3186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:09.710588 3186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98665c3e030c44768014c0c1872c315a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35041 } }
I20260925 05:10:09.710716 3186 leader_election.cc:302] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98665c3e030c44768014c0c1872c315a; no voters:
I20260925 05:10:09.710912 3186 leader_election.cc:288] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:09.710947 3191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:09.711113 3191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [term 1 LEADER]: Becoming Leader. State: Replica: 98665c3e030c44768014c0c1872c315a, State: Running, Role: LEADER
I20260925 05:10:09.711211 3186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:09.711221 3191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98665c3e030c44768014c0c1872c315a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35041 } }
I20260925 05:10:09.711898 3192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98665c3e030c44768014c0c1872c315a. Latest consensus state: current_term: 1 leader_uuid: "98665c3e030c44768014c0c1872c315a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98665c3e030c44768014c0c1872c315a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35041 } } }
I20260925 05:10:09.711990 3192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:09.712117 3193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98665c3e030c44768014c0c1872c315a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98665c3e030c44768014c0c1872c315a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35041 } } }
I20260925 05:10:09.712177 3193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:09.712450 3198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:09.712989 3198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:09.715024 3198 catalog_manager.cc:1347] Generated new cluster ID: 2d82a533389d47a08e6a935dafc27edd
I20260925 05:10:09.715085 3198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:10:09 dist-test-slave-1lvv krb5kdc[3105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:09.723456 3198 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:09.724223 3198 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:09.726811 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.708865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.3.6.190:35041)
0925 05:10:09.709374 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.709383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.709489 (+ 106us) server_negotiation.cc:407] Connection header received
0925 05:10:09.712278 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.712286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.712357 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.712523 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.715380 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.716776 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.717988 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.718529 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.718720 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.722653 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.722658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.722658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.722672 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.724792 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.725253 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.725257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.725257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.725370 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.725683 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.725684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.725684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.725919 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.726112 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.726381 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.726542 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":459,"thread_start_us":116,"threads_started":1}
I20260925 05:10:09.730050 3198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98665c3e030c44768014c0c1872c315a: Generated new TSK 0
I20260925 05:10:09.730269 3198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:09.767479 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:35041
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:09.856563 3212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:09.856792 3212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:09.856818 3212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:09.857924 3212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:09.857957 3212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:09.857983 3212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:09.859352 3212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:35041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:09.859550 3212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:09.859735 3212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:09.864894 3220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:09.865037 3212 server_base.cc:1034] running on GCE node
W20260925 05:10:09.864889 3218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:09.864970 3217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:09.865350 3212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:09.865621 3212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:09.866809 3212 hybrid_clock.cc:648] HybridClock initialized: now 1790313009866790 us; error 30 us; skew 500 ppm
Sep 25 05:10:09 dist-test-slave-1lvv krb5kdc[3105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:09.869372 3212 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:09.869889 3212 webserver.cc:460] Webserver started at http://127.3.6.129:34311/ using document root <none> and password file <none>
I20260925 05:10:09.870118 3212 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:09.870189 3212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:09.870335 3212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:09.871459 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "4db4ec3c313a4c39be9698dd00d25c02"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-1lvv"
server_key: "cad7624e42d156f1456b31948adc9abf"
server_key_iv: "246c1d04450c20d4f8fa987f20cb06d9"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.871894 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "4db4ec3c313a4c39be9698dd00d25c02"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-1lvv"
server_key: "cad7624e42d156f1456b31948adc9abf"
server_key_iv: "246c1d04450c20d4f8fa987f20cb06d9"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.873413 3212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:09.874137 3227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:09.874327 3212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:09.874433 3212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "4db4ec3c313a4c39be9698dd00d25c02"
format_stamp: "Formatted at 2026-09-25 05:10:09 on dist-test-slave-1lvv"
server_key: "cad7624e42d156f1456b31948adc9abf"
server_key_iv: "246c1d04450c20d4f8fa987f20cb06d9"
server_key_version: "encryptionkey@0"
I20260925 05:10:09.874511 3212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:09.907666 3212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:09.909739 3212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:09.909909 3212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:09.910162 3212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:09.910488 3212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:09.910543 3212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:09.910610 3212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:09.910635 3212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:09.916736 3212 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:37947
I20260925 05:10:09.916796 3340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:37947 every 8 connection(s)
I20260925 05:10:09.917152 3212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:09 dist-test-slave-1lvv krb5kdc[3105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:09.925246 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3212
I20260925 05:10:09.925354 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:09.925534 3098 external_mini_cluster.cc:1428] Setting key e0fd486468fb7cdb6f411bbea0f6b095
I20260925 05:10:09.927096 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.917897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:35041 (local address 127.3.6.129:35959)
0925 05:10:09.918090 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:09.918127 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:09.918637 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:09.918930 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:09.918934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:09.919646 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:09.920070 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.920077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.920805 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.920806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.921613 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:09.921615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.922030 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.922031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:09.922153 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.922651 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:09.922659 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:09.924030 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:09.925582 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.925591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.925593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.925828 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.926019 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:09.926020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:09.926020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:09.926176 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:09.926602 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:09.926607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:09.926727 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 05:10:09.926809 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:10:09.926930 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260925 05:10:09.927098 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:09.917930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:35959 (local address 127.3.6.190:35041)
0925 05:10:09.917999 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:09.918001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:09.918217 (+ 216us) server_negotiation.cc:407] Connection header received
0925 05:10:09.918764 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:09.918766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:09.918778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:09.918834 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:09.920188 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.920705 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.921726 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:09.921935 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:09.922018 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:09.924128 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:09.924130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:09.924131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:09.924145 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:09.925492 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.925896 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.925897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.925897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.925959 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:09.926356 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:09.926357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:09.926358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:09.926506 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:09.926578 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:09.926850 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 05:10:09.926952 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:10:09.927635 3341 heartbeater.cc:347] Connected to a master server at 127.3.6.190:35041
I20260925 05:10:09.927744 3341 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:09.927973 3341 heartbeater.cc:511] Master 127.3.6.190:35041 requested a full tablet report, sending...
I20260925 05:10:09.928656 3150 ts_manager.cc:194] Registered new tserver with Master: 4db4ec3c313a4c39be9698dd00d25c02 (127.3.6.129:37947)
I20260925 05:10:09.929498 3150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:35959
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:09.959614 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:35041
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:10.053508 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:10.053763 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:10.053805 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:10.054970 3346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:10.055037 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:10.055083 3346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:10.056576 3346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:35041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:10.056850 3346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:10.057080 3346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:10.062156 3352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:10.062171 3351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:10.062400 3346 server_base.cc:1034] running on GCE node
W20260925 05:10:10.062194 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:10.062666 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:10.062873 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:10.064051 3346 hybrid_clock.cc:648] HybridClock initialized: now 1790313010064036 us; error 31 us; skew 500 ppm
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:10.066596 3346 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:10.067103 3346 webserver.cc:460] Webserver started at http://127.3.6.130:45371/ using document root <none> and password file <none>
I20260925 05:10:10.067353 3346 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:10.067399 3346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:10.067553 3346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:10.068804 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "6359f0bd20ea40c4a33c9e9ea431a555"
format_stamp: "Formatted at 2026-09-25 05:10:10 on dist-test-slave-1lvv"
server_key: "6fa6eee8fb5718c0cd614fe30d0234e1"
server_key_iv: "33a72f4ff4a5ecea4f9fb906405119c5"
server_key_version: "encryptionkey@0"
I20260925 05:10:10.069192 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "6359f0bd20ea40c4a33c9e9ea431a555"
format_stamp: "Formatted at 2026-09-25 05:10:10 on dist-test-slave-1lvv"
server_key: "6fa6eee8fb5718c0cd614fe30d0234e1"
server_key_iv: "33a72f4ff4a5ecea4f9fb906405119c5"
server_key_version: "encryptionkey@0"
I20260925 05:10:10.070729 3346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:10.071462 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:10.071643 3346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:10.071734 3346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "6359f0bd20ea40c4a33c9e9ea431a555"
format_stamp: "Formatted at 2026-09-25 05:10:10 on dist-test-slave-1lvv"
server_key: "6fa6eee8fb5718c0cd614fe30d0234e1"
server_key_iv: "33a72f4ff4a5ecea4f9fb906405119c5"
server_key_version: "encryptionkey@0"
I20260925 05:10:10.071833 3346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:10.082247 3346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:10.084203 3346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:10.084367 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:10.084600 3346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:10.084941 3346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:10.084993 3346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:10.085028 3346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:10.085055 3346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:10.091325 3346 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:42419
I20260925 05:10:10.091372 3474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:42419 every 8 connection(s)
I20260925 05:10:10.091717 3346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:10.095531 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3346
I20260925 05:10:10.095614 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:10.095852 3098 external_mini_cluster.cc:1428] Setting key 458cc4c2d17d32eae74b65c927281ecb
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:10.101469 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:10.092538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:40437 (local address 127.3.6.190:35041)
0925 05:10:10.092620 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:10.092623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:10.092778 (+ 155us) server_negotiation.cc:407] Connection header received
0925 05:10:10.093263 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:10.093264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:10.093274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:10.093331 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:10.094290 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.094775 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.095856 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.096138 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.096219 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:10.098599 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:10.098600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:10.098601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:10.098609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:10.100095 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.100509 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.100510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.100511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.100568 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.100820 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.100821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.100821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.100934 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:10.101002 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:10.101284 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 05:10:10.101375 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:10:10.101665 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:10.092499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:35041 (local address 127.3.6.130:40437)
0925 05:10:10.092658 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:10.092694 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:10.093179 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:10.093385 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:10.093389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:10.093950 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:10.094216 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:10.094221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.094857 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.094858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:10.095741 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:10.095743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.096233 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.096234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:10.096374 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:10.096921 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:10.096929 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:10.098502 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:10.100196 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:10.100202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:10.100205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:10.100425 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:10.100636 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:10.100637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:10.100637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:10.100749 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:10.101004 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:10.101011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:10.101150 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 05:10:10.101262 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 05:10:10.101428 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260925 05:10:10.102171 3475 heartbeater.cc:347] Connected to a master server at 127.3.6.190:35041
I20260925 05:10:10.102265 3475 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:10.102460 3475 heartbeater.cc:511] Master 127.3.6.190:35041 requested a full tablet report, sending...
I20260925 05:10:10.102859 3150 ts_manager.cc:194] Registered new tserver with Master: 6359f0bd20ea40c4a33c9e9ea431a555 (127.3.6.130:42419)
I20260925 05:10:10.103401 3150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:40437
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:10.123646 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:35041
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:10.206483 3480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:10.206745 3480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:10.206785 3480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:10.208004 3480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:10.208076 3480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:10.208122 3480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:10.209625 3480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:35041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:10.209905 3480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:10.210145 3480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:10.215694 3485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:10.215694 3486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:10.215937 3480 server_base.cc:1034] running on GCE node
W20260925 05:10:10.215737 3488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:10.216225 3480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:10.216435 3480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:10.217638 3480 hybrid_clock.cc:648] HybridClock initialized: now 1790313010217626 us; error 30 us; skew 500 ppm
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:10.220247 3480 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:10.220686 3480 webserver.cc:460] Webserver started at http://127.3.6.131:36771/ using document root <none> and password file <none>
I20260925 05:10:10.220911 3480 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:10.220985 3480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:10.221130 3480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:10.222223 3480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "b5643645319b4b638d43cb720f3709fc"
format_stamp: "Formatted at 2026-09-25 05:10:10 on dist-test-slave-1lvv"
server_key: "88a4803050bc8d4cb242da79cb8ebd7f"
server_key_iv: "7a7dd5385e5681c71ac7be06f9954ef8"
server_key_version: "encryptionkey@0"
I20260925 05:10:10.222612 3480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "b5643645319b4b638d43cb720f3709fc"
format_stamp: "Formatted at 2026-09-25 05:10:10 on dist-test-slave-1lvv"
server_key: "88a4803050bc8d4cb242da79cb8ebd7f"
server_key_iv: "7a7dd5385e5681c71ac7be06f9954ef8"
server_key_version: "encryptionkey@0"
I20260925 05:10:10.224079 3480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 05:10:10.224772 3495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:10.224933 3480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:10.225033 3480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "b5643645319b4b638d43cb720f3709fc"
format_stamp: "Formatted at 2026-09-25 05:10:10 on dist-test-slave-1lvv"
server_key: "88a4803050bc8d4cb242da79cb8ebd7f"
server_key_iv: "7a7dd5385e5681c71ac7be06f9954ef8"
server_key_version: "encryptionkey@0"
I20260925 05:10:10.225103 3480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:10.244094 3480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:10.245991 3480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:10.246160 3480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:10.246402 3480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:10.246711 3480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:10.246762 3480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:10.246804 3480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:10.246843 3480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:10.253073 3480 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:46251
I20260925 05:10:10.253149 3608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:46251 every 8 connection(s)
I20260925 05:10:10.253500 3480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:10.259209 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3480
I20260925 05:10:10.259302 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:10.259510 3098 external_mini_cluster.cc:1428] Setting key a28eaa1a7a96a7669868f053e1a49755
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:10.263326 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:10.254361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:39925 (local address 127.3.6.190:35041)
0925 05:10:10.254438 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:10.254441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:10.254693 (+ 252us) server_negotiation.cc:407] Connection header received
0925 05:10:10.255174 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:10.255175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:10.255186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:10.255248 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:10.256278 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.256774 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.257731 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.257911 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.257967 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:10.260538 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:10.260539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:10.260539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:10.260546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:10.262123 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.262516 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.262516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.262517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.262575 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.262807 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.262808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.262808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.262927 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:10.262991 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:10.263200 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 05:10:10.263263 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:10:10.263514 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:10.254324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:35041 (local address 127.3.6.131:39925)
0925 05:10:10.254547 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:10.254586 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:10.255075 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:10.255304 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:10.255307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:10.255988 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:10.256191 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:10.256196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.256855 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.256856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:10.257655 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:10.257657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.257982 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.257983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:10.258078 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:10.258655 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:10.258663 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:10.260453 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:10.262201 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:10.262207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:10.262209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:10.262426 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:10.262634 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:10.262635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:10.262635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:10.262729 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:10.262989 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:10.262993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:10.263105 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 05:10:10.263185 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:10:10.263303 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260925 05:10:10.264041 3609 heartbeater.cc:347] Connected to a master server at 127.3.6.190:35041
I20260925 05:10:10.264134 3609 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:10.264326 3609 heartbeater.cc:511] Master 127.3.6.190:35041 requested a full tablet report, sending...
I20260925 05:10:10.264688 3150 ts_manager.cc:194] Registered new tserver with Master: b5643645319b4b638d43cb720f3709fc (127.3.6.131:46251)
I20260925 05:10:10.265013 3150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:39925
I20260925 05:10:10.271878 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:10.296229 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:10.288591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.3.6.129:37947)
0925 05:10:10.288737 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:10.288748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:10.288761 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:10:10.288790 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:10.288792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:10.288806 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:10.288873 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:10.289417 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.290162 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.290883 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.291224 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.291286 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:10.293653 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:10.293664 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:10.293664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:10.293671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:10.295102 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.295415 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.295417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.295418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.295492 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.295654 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.295655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.295655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.295862 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:10.295985 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:10.296042 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 05:10:10.296110 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260925 05:10:10.296610 3255 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44298
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:10.320982 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:10.313991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.3.6.129:37947)
0925 05:10:10.314071 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:10.314074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:10.314120 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:10:10.314181 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:10.314182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:10.314194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:10.314247 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:10.314808 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.315298 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.316100 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:10.316307 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:10.316367 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:10.318630 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:10.318631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:10.318632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:10.318640 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:10.319972 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.320286 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.320287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.320287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.320338 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:10.320511 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:10.320511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:10.320512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:10.320606 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:10.320699 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:10.320780 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 05:10:10.320864 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:10:10.321554 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3212
I20260925 05:10:10.327519 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3346
I20260925 05:10:10.333238 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3480
I20260925 05:10:10.338150 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3118
2026-09-25T05:10:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1974 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3623](info): set up 2 sockets
Sep 25 05:10:10 dist-test-slave-1lvv krb5kdc[3623](info): commencing operation
krb5kdc: starting...
W20260925 05:10:11.375500 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:11 dist-test-slave-1lvv krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:11.507354 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45753
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:45753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:11.590670 3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:11.590947 3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:11.590986 3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:11.592175 3636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:11.592247 3636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:11.592271 3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:11.592298 3636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:11.592330 3636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:11.593855 3636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:45753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:11.594149 3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:11.594384 3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:11.599274 3641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:11.599274 3644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:11.599368 3642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:11.599633 3636 server_base.cc:1034] running on GCE node
I20260925 05:10:11.599949 3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:11.600281 3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:11.601475 3636 hybrid_clock.cc:648] HybridClock initialized: now 1790313011601457 us; error 28 us; skew 500 ppm
Sep 25 05:10:11 dist-test-slave-1lvv krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:11.604308 3636 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:11.604885 3636 webserver.cc:460] Webserver started at http://127.3.6.190:38731/ using document root <none> and password file <none>
I20260925 05:10:11.605140 3636 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:11.605209 3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:11.605352 3636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:11.606475 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "7fd332d766184b4c89b39d908d1b1c2c"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "dbb6e28c5cf20243cedee400d0feda45"
server_key_iv: "cb37aea79f6bc975006e4b38c845503f"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.606868 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "7fd332d766184b4c89b39d908d1b1c2c"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "dbb6e28c5cf20243cedee400d0feda45"
server_key_iv: "cb37aea79f6bc975006e4b38c845503f"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.608421 3636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:11.609161 3651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.609329 3636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:11.609434 3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "7fd332d766184b4c89b39d908d1b1c2c"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "dbb6e28c5cf20243cedee400d0feda45"
server_key_iv: "cb37aea79f6bc975006e4b38c845503f"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.609505 3636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:11.617926 3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:11.619930 3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:11.620110 3636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:11.624221 3636 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:45753
I20260925 05:10:11.624343 3703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:45753 every 8 connection(s)
I20260925 05:10:11.624672 3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:11.625631 3704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:11.628257 3704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c: Bootstrap starting.
I20260925 05:10:11.628891 3704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:11.629169 3704 log.cc:824] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:11.629835 3704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c: No bootstrap required, opened a new log
I20260925 05:10:11.632439 3704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd332d766184b4c89b39d908d1b1c2c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45753 } }
I20260925 05:10:11.632683 3704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:11.632740 3704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd332d766184b4c89b39d908d1b1c2c, State: Initialized, Role: FOLLOWER
I20260925 05:10:11.632910 3704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd332d766184b4c89b39d908d1b1c2c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45753 } }
I20260925 05:10:11.633025 3704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:11.633068 3704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:11.633092 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3636
I20260925 05:10:11.633128 3704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:11.633178 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:11.633363 3098 external_mini_cluster.cc:1428] Setting key f19cc8a676d82869e4f4ce2afad4f06f
I20260925 05:10:11.633929 3704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd332d766184b4c89b39d908d1b1c2c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45753 } }
I20260925 05:10:11.634055 3704 leader_election.cc:302] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd332d766184b4c89b39d908d1b1c2c; no voters:
I20260925 05:10:11.634225 3704 leader_election.cc:288] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:11.634522 3704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:11.634680 3709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:11.634826 3709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [term 1 LEADER]: Becoming Leader. State: Replica: 7fd332d766184b4c89b39d908d1b1c2c, State: Running, Role: LEADER
I20260925 05:10:11.634919 3709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd332d766184b4c89b39d908d1b1c2c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45753 } }
I20260925 05:10:11.635313 3709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd332d766184b4c89b39d908d1b1c2c. Latest consensus state: current_term: 1 leader_uuid: "7fd332d766184b4c89b39d908d1b1c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd332d766184b4c89b39d908d1b1c2c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45753 } } }
I20260925 05:10:11.635401 3709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:11.635347 3710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd332d766184b4c89b39d908d1b1c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd332d766184b4c89b39d908d1b1c2c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45753 } } }
I20260925 05:10:11.635624 3718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:11.635654 3710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:11.636267 3718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:11.638669 3718 catalog_manager.cc:1347] Generated new cluster ID: e19544dcc8e24c329057a682c1d2e2ff
I20260925 05:10:11.638731 3718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:10:11 dist-test-slave-1lvv krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:11.648744 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:11.634045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.3.6.190:45753)
0925 05:10:11.634295 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:11.634306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:11.636474 (+ 2168us) server_negotiation.cc:407] Connection header received
0925 05:10:11.636700 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:11.636709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:11.636791 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:11.636967 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:11.638435 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.639948 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.640985 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.641593 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.641666 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:11.644873 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:11.644879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:11.644879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:11.644893 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:11.646905 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:11.647376 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:11.647379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:11.647379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:11.647489 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:11.647716 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:11.647716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:11.647717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:11.648065 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:11.648229 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:11.648367 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:10:11.648508 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":130,"threads_started":1}
I20260925 05:10:11.649541 3718 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:11.650041 3718 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:11.656644 3718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fd332d766184b4c89b39d908d1b1c2c: Generated new TSK 0
I20260925 05:10:11.656853 3718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:11.687067 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45753
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:11.773761 3730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:11.773999 3730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:11.774027 3730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:11.775149 3730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:11.775194 3730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:11.775224 3730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:11.776798 3730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:11.777014 3730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:11.777209 3730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:11.782594 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:11.782645 3738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:11.782603 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:11.782842 3730 server_base.cc:1034] running on GCE node
I20260925 05:10:11.783126 3730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:11.783399 3730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:11.784621 3730 hybrid_clock.cc:648] HybridClock initialized: now 1790313011784604 us; error 34 us; skew 500 ppm
Sep 25 05:10:11 dist-test-slave-1lvv krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:11.787483 3730 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:11.788122 3730 webserver.cc:460] Webserver started at http://127.3.6.129:45645/ using document root <none> and password file <none>
I20260925 05:10:11.788396 3730 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:11.788466 3730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:11.788621 3730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:11.789731 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "5a5b29575aee4c2081b1b7483a6c6e9d"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "9f00fbfdd173689b06b3c34130b52007"
server_key_iv: "62309a6fc04b5c1d5b78da5a1854e452"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.790160 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "5a5b29575aee4c2081b1b7483a6c6e9d"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "9f00fbfdd173689b06b3c34130b52007"
server_key_iv: "62309a6fc04b5c1d5b78da5a1854e452"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.791525 3730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:11.792382 3745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.792563 3730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.792650 3730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "5a5b29575aee4c2081b1b7483a6c6e9d"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "9f00fbfdd173689b06b3c34130b52007"
server_key_iv: "62309a6fc04b5c1d5b78da5a1854e452"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.792742 3730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:11.806056 3730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:11.808254 3730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:11.808373 3730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:11.808563 3730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:11.808879 3730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:11.808914 3730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.808957 3730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:11.808983 3730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.814857 3730 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:46161
I20260925 05:10:11.814932 3858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:46161 every 8 connection(s)
I20260925 05:10:11.815267 3730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:11 dist-test-slave-1lvv krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:11.822958 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3730
I20260925 05:10:11.823081 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:11.823256 3098 external_mini_cluster.cc:1428] Setting key b52ad1d7fb5942b12c99e96b1a9f0a2d
I20260925 05:10:11.825947 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:11.816124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:37049 (local address 127.3.6.190:45753)
0925 05:10:11.816208 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:11.816211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:11.816431 (+ 220us) server_negotiation.cc:407] Connection header received
0925 05:10:11.817543 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:11.817544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:11.817559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:11.817617 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:11.818261 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.818768 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.820019 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.820229 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.820288 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:11.822472 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:11.822474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:11.822474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:11.822485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:11.824383 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:11.824825 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:11.824826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:11.824827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:11.824887 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:11.825146 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:11.825147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:11.825147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:11.825305 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:11.825380 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:11.825714 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 05:10:11.825822 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:10:11.825966 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:11.816112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45753 (local address 127.3.6.129:37049)
0925 05:10:11.816294 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:11.816332 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:11.817437 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:11.817678 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:11.817681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:11.817961 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:11.818170 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:11.818175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.818870 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.818871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:11.819928 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:11.819931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.820320 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.820321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:11.820427 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:11.820940 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:11.820949 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:11.822384 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:11.824485 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:11.824493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:11.824495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:11.824731 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:11.824953 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:11.824954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:11.824954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:11.825069 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:11.825389 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:11.825397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:11.825565 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 05:10:11.825662 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:10:11.825797 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260925 05:10:11.826505 3859 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45753
I20260925 05:10:11.826620 3859 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:11.826803 3859 heartbeater.cc:511] Master 127.3.6.190:45753 requested a full tablet report, sending...
I20260925 05:10:11.827396 3668 ts_manager.cc:194] Registered new tserver with Master: 5a5b29575aee4c2081b1b7483a6c6e9d (127.3.6.129:46161)
I20260925 05:10:11.828420 3668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:37049
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:11.850750 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45753
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:11.933023 3864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:11.933255 3864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:11.933282 3864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:11.934419 3864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:11.934468 3864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:11.934499 3864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:11.936336 3864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:11.936590 3864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:11.936800 3864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:11.942164 3870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:11.942122 3872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:11.942122 3869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:11.942420 3864 server_base.cc:1034] running on GCE node
I20260925 05:10:11.942672 3864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:11.942932 3864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:11.944096 3864 hybrid_clock.cc:648] HybridClock initialized: now 1790313011944079 us; error 42 us; skew 500 ppm
Sep 25 05:10:11 dist-test-slave-1lvv krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:11.946852 3864 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:11.947378 3864 webserver.cc:460] Webserver started at http://127.3.6.130:42851/ using document root <none> and password file <none>
I20260925 05:10:11.947616 3864 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:11.947690 3864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:11.947868 3864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:11.949103 3864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "89f29459b0684d91bf12b2698cf3c3b4"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "8a11f8e02adbdc51a6cba996c589a373"
server_key_iv: "133f892cd39906ae49a5f17cc09dfff2"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.949491 3864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "89f29459b0684d91bf12b2698cf3c3b4"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "8a11f8e02adbdc51a6cba996c589a373"
server_key_iv: "133f892cd39906ae49a5f17cc09dfff2"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.950958 3864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:11.951893 3879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.952086 3864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:11.952190 3864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "89f29459b0684d91bf12b2698cf3c3b4"
format_stamp: "Formatted at 2026-09-25 05:10:11 on dist-test-slave-1lvv"
server_key: "8a11f8e02adbdc51a6cba996c589a373"
server_key_iv: "133f892cd39906ae49a5f17cc09dfff2"
server_key_version: "encryptionkey@0"
I20260925 05:10:11.952279 3864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:11.968945 3864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:11.971596 3864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:11.971838 3864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:11.972108 3864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:11.972440 3864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:11.972492 3864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.972528 3864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:11.972571 3864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:11.978688 3864 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:44745
I20260925 05:10:11.978756 3992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:44745 every 8 connection(s)
I20260925 05:10:11.979068 3864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:10:11 dist-test-slave-1lvv krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:11.986397 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3864
I20260925 05:10:11.986552 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:11.986757 3098 external_mini_cluster.cc:1428] Setting key a03bd2ca00f1f67b8ce183bcefa38959
I20260925 05:10:11.988981 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:11.979864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:40973 (local address 127.3.6.190:45753)
0925 05:10:11.979928 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:11.979930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:11.980165 (+ 235us) server_negotiation.cc:407] Connection header received
0925 05:10:11.981080 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:11.981081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:11.981092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:11.981147 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:11.981737 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.982225 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.983184 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.983364 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.983414 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:11.985536 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:11.985537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:11.985538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:11.985545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:11.987141 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:11.987733 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:11.987734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:11.987734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:11.987823 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:11.988094 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:11.988095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:11.988095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:11.988266 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:11.988366 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:11.988733 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 05:10:11.988835 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:10:11.989019 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:11.979872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45753 (local address 127.3.6.130:40973)
0925 05:10:11.980028 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:11.980066 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:11.980993 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:11.981200 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:11.981204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:11.981464 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:11.981668 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:11.981672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.982297 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.982298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:11.983114 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:11.983116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:11.983428 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:11.983428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:11.983529 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:11.984038 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:11.984047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:11.985435 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:11.987263 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:11.987273 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:11.987276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:11.987655 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:11.987908 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:11.987909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:11.987909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:11.988031 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:11.988423 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:11.988431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:11.988587 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 05:10:11.988690 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:10:11.988835 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260925 05:10:11.989691 3993 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45753
I20260925 05:10:11.989796 3993 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:11.989986 3993 heartbeater.cc:511] Master 127.3.6.190:45753 requested a full tablet report, sending...
I20260925 05:10:11.990417 3668 ts_manager.cc:194] Registered new tserver with Master: 89f29459b0684d91bf12b2698cf3c3b4 (127.3.6.130:44745)
I20260925 05:10:11.990809 3668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:40973
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:12.014117 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45753
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:12.099020 3998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:12.099284 3998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:12.099326 3998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:12.100914 3998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:12.100996 3998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:12.101048 3998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:12.102622 3998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:12.102914 3998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:12.103193 3998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:12.108582 4003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:12.108534 4006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:12.108530 4004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:12.108764 3998 server_base.cc:1034] running on GCE node
I20260925 05:10:12.109004 3998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:12.109231 3998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:12.110378 3998 hybrid_clock.cc:648] HybridClock initialized: now 1790313012110365 us; error 27 us; skew 500 ppm
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:12.113119 3998 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:12.113655 3998 webserver.cc:460] Webserver started at http://127.3.6.131:44025/ using document root <none> and password file <none>
I20260925 05:10:12.113888 3998 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:12.113955 3998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:12.114109 3998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:12.115187 3998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "ba606fa79a284245b41276d5c312c023"
format_stamp: "Formatted at 2026-09-25 05:10:12 on dist-test-slave-1lvv"
server_key: "a634f1947bc4f1215ba92e3e00105f36"
server_key_iv: "f53552ec2905ba3c52b68b59363e5659"
server_key_version: "encryptionkey@0"
I20260925 05:10:12.115561 3998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "ba606fa79a284245b41276d5c312c023"
format_stamp: "Formatted at 2026-09-25 05:10:12 on dist-test-slave-1lvv"
server_key: "a634f1947bc4f1215ba92e3e00105f36"
server_key_iv: "f53552ec2905ba3c52b68b59363e5659"
server_key_version: "encryptionkey@0"
I20260925 05:10:12.117250 3998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:12.118116 4013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:12.118321 3998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:12.118433 3998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "ba606fa79a284245b41276d5c312c023"
format_stamp: "Formatted at 2026-09-25 05:10:12 on dist-test-slave-1lvv"
server_key: "a634f1947bc4f1215ba92e3e00105f36"
server_key_iv: "f53552ec2905ba3c52b68b59363e5659"
server_key_version: "encryptionkey@0"
I20260925 05:10:12.118546 3998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:12.128576 3998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:12.130627 3998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:12.130806 3998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:12.131055 3998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:12.131392 3998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:12.131448 3998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:12.131500 3998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:12.131547 3998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:12.138689 3998 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:34339
I20260925 05:10:12.138731 4126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:34339 every 8 connection(s)
I20260925 05:10:12.139115 3998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:12.139482 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 3998
I20260925 05:10:12.139595 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:12.139861 3098 external_mini_cluster.cc:1428] Setting key 8c1edbbe51eedb0b718304142a3a751c
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:12.148561 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.140023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:54755 (local address 127.3.6.190:45753)
0925 05:10:12.140077 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.140079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.140468 (+ 389us) server_negotiation.cc:407] Connection header received
0925 05:10:12.141525 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.141526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.141537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.141589 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:12.142188 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.142691 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.143659 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.143931 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.143985 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.146108 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:12.146109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:12.146109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:12.146116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:12.147357 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.147716 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.147717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.147717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.147812 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.148030 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.148031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.148031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.148149 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:12.148202 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.148425 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.148494 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:10:12.148809 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.140010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45753 (local address 127.3.6.131:54755)
0925 05:10:12.140256 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:12.140302 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:12.141430 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:12.141647 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:12.141651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:12.141919 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:12.142118 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.142122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.142769 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.142770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.143576 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.143578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.144009 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.144010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.144111 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.144572 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:12.144581 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:12.146020 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:12.147426 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:12.147432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:12.147433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:12.147648 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:12.147875 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:12.147875 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:12.147876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:12.147976 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:12.148205 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:12.148209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:12.148319 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 05:10:12.148399 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:10:12.148517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260925 05:10:12.149281 4127 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45753
I20260925 05:10:12.149398 4127 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:12.149583 4127 heartbeater.cc:511] Master 127.3.6.190:45753 requested a full tablet report, sending...
I20260925 05:10:12.149945 3668 ts_manager.cc:194] Registered new tserver with Master: ba606fa79a284245b41276d5c312c023 (127.3.6.131:34339)
I20260925 05:10:12.150296 3668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:54755
I20260925 05:10:12.151718 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:12.159358 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.153366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.3.6.190:45753)
0925 05:10:12.153443 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.153446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.153531 (+ 85us) server_negotiation.cc:407] Connection header received
0925 05:10:12.153639 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.153640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.153651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.153701 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:12.154252 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.154786 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.155543 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.155738 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.155865 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.157123 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:12.157125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:12.157125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:12.157133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:12.158368 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.158693 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.158694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.158695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.158751 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.158931 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.158931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.158932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.159068 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:12.159147 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.159231 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.159294 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:10:12.161440 3668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:12.163017 3668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:12.169909 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.165491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:46161 (local address 127.0.0.1:54148)
0925 05:10:12.165886 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:12.165899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:12.165950 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:12.166482 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:12.166483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:12.166485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:12.166527 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.166529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.167402 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.167403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.168577 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.168580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.169564 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.169565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.169649 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.169664 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 05:10:12.169752 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:10:12.169809 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":44,"thread_start_us":128,"threads_started":1}
I20260925 05:10:12.169890 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.165827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.3.6.129:46161)
0925 05:10:12.166240 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.166245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.166256 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:12.166294 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.166296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.166311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.166388 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:12.166611 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.167293 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.168664 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.169472 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.169557 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.169622 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.169785 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.169823 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":86,"threads_started":1}
I20260925 05:10:12.170223 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.165767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:34339 (local address 127.0.0.1:34712)
0925 05:10:12.166625 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:12.166634 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:12.166673 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:12.166936 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:12.166936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:12.166938 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:12.166974 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.166975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.168261 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.168263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.169152 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.169154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.169956 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.169958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.170026 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.170032 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:10:12.170098 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 05:10:12.170150 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":83,"threads_started":1}
I20260925 05:10:12.170552 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.166021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.3.6.131:34339)
0925 05:10:12.166260 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.166264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.166738 (+ 474us) server_negotiation.cc:407] Connection header received
0925 05:10:12.166781 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.166783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.166798 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.166859 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:12.167042 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.168146 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.169237 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.169870 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.169942 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.170000 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.170408 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.170457 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260925 05:10:12.170847 3793 tablet_service.cc:1467] Processing CreateTablet for tablet 52e8b47899634ff5bee5009056ac31c3 (DEFAULT_TABLE table=test-table [id=503b951290ac4c88976b009a9e2397d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:12.171208 3793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e8b47899634ff5bee5009056ac31c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:12.171208 4061 tablet_service.cc:1467] Processing CreateTablet for tablet 52e8b47899634ff5bee5009056ac31c3 (DEFAULT_TABLE table=test-table [id=503b951290ac4c88976b009a9e2397d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:12.171537 4061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e8b47899634ff5bee5009056ac31c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:12.172052 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.165491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:44745 (local address 127.0.0.1:49224)
0925 05:10:12.165694 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:12.165740 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:12.165815 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:12.166060 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:12.166062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:12.166064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:12.166147 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.166149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.167528 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.167528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.168690 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.168693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.170146 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.170146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.171863 (+ 1717us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.171870 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:10:12.171941 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:10:12.171983 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":52,"threads_started":1}
I20260925 05:10:12.172099 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.165604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.3.6.130:44745)
0925 05:10:12.165785 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.165791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.165804 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:10:12.165947 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.165950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.165966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.166057 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:12.166291 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.167408 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.168777 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.169473 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.169557 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.169621 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.171968 (+ 2347us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.172023 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260925 05:10:12.173063 3927 tablet_service.cc:1467] Processing CreateTablet for tablet 52e8b47899634ff5bee5009056ac31c3 (DEFAULT_TABLE table=test-table [id=503b951290ac4c88976b009a9e2397d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:12.173406 3927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e8b47899634ff5bee5009056ac31c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:12.174539 4144 tablet_bootstrap.cc:492] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023: Bootstrap starting.
I20260925 05:10:12.175261 4144 tablet_bootstrap.cc:654] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:12.175714 4144 log.cc:824] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:12.176187 4145 tablet_bootstrap.cc:492] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d: Bootstrap starting.
I20260925 05:10:12.176620 4147 tablet_bootstrap.cc:492] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4: Bootstrap starting.
I20260925 05:10:12.176693 4144 tablet_bootstrap.cc:492] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023: No bootstrap required, opened a new log
I20260925 05:10:12.176767 4144 ts_tablet_manager.cc:1397] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:12.176939 4145 tablet_bootstrap.cc:654] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:12.177294 4147 tablet_bootstrap.cc:654] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:12.177516 4145 log.cc:824] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:12.177865 4147 log.cc:824] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:12.178260 4145 tablet_bootstrap.cc:492] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d: No bootstrap required, opened a new log
I20260925 05:10:12.178345 4145 ts_tablet_manager.cc:1397] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:12.178647 4147 tablet_bootstrap.cc:492] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4: No bootstrap required, opened a new log
I20260925 05:10:12.178718 4147 ts_tablet_manager.cc:1397] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:12.178922 4144 raft_consensus.cc:348] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.179141 4144 raft_consensus.cc:374] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:12.179188 4144 raft_consensus.cc:729] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba606fa79a284245b41276d5c312c023, State: Initialized, Role: FOLLOWER
I20260925 05:10:12.179342 4144 consensus_queue.cc:260] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.179667 4144 ts_tablet_manager.cc:1428] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:12.179841 4127 heartbeater.cc:503] Master 127.3.6.190:45753 was elected leader, sending a full tablet report...
I20260925 05:10:12.180675 4147 raft_consensus.cc:348] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.180864 4147 raft_consensus.cc:374] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:12.180904 4147 raft_consensus.cc:729] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f29459b0684d91bf12b2698cf3c3b4, State: Initialized, Role: FOLLOWER
I20260925 05:10:12.181021 4147 consensus_queue.cc:260] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.181007 4145 raft_consensus.cc:348] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.181180 4145 raft_consensus.cc:374] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:12.181231 4145 raft_consensus.cc:729] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5b29575aee4c2081b1b7483a6c6e9d, State: Initialized, Role: FOLLOWER
I20260925 05:10:12.181309 4147 ts_tablet_manager.cc:1428] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:12.181344 4145 consensus_queue.cc:260] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.181452 3993 heartbeater.cc:503] Master 127.3.6.190:45753 was elected leader, sending a full tablet report...
I20260925 05:10:12.181648 4145 ts_tablet_manager.cc:1428] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:12.181806 3859 heartbeater.cc:503] Master 127.3.6.190:45753 was elected leader, sending a full tablet report...
I20260925 05:10:12.226897 4152 raft_consensus.cc:482] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:12.227020 4152 raft_consensus.cc:504] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.227460 4152 leader_election.cc:288] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f29459b0684d91bf12b2698cf3c3b4 (127.3.6.130:44745), ba606fa79a284245b41276d5c312c023 (127.3.6.131:34339)
W20260925 05:10:12.229605 3994 tablet.cc:2369] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:12.231520 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.227870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:34293 (local address 127.3.6.131:34339)
0925 05:10:12.227988 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.227990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.228140 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:10:12.228169 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.228169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.228188 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.228237 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:12.228401 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.229076 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.230573 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.231145 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.231229 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.231238 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.231411 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.231453 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 05:10:12.231520 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.227865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:42575 (local address 127.3.6.130:44745)
0925 05:10:12.227933 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.227935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.227947 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:12.228034 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.228035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.228046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.228090 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:12.228291 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.229065 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.230573 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.231145 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.231229 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.231238 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.231410 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.231453 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:10:12.231555 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.227636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:44745 (local address 127.3.6.129:42575)
0925 05:10:12.227696 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:12.227710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:12.227933 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:12.228118 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:12.228120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:12.228122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:12.228206 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.228207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.229153 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.229154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.230471 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.230474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.231243 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.231244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.231334 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.231336 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:12.231395 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:10:12.231452 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":89}
I20260925 05:10:12.231559 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.227746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:34339 (local address 127.3.6.129:34293)
0925 05:10:12.228056 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:12.228063 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:12.228124 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:12.228260 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:12.228261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:12.228263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:12.228329 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.228330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.229153 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.229154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.230471 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:12.230474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.231243 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.231244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:12.231330 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.231334 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:12.231395 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:10:12.231452 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":53,"thread_start_us":156,"threads_started":1}
I20260925 05:10:12.232007 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e8b47899634ff5bee5009056ac31c3" candidate_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba606fa79a284245b41276d5c312c023" is_pre_election: true
I20260925 05:10:12.232052 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e8b47899634ff5bee5009056ac31c3" candidate_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f29459b0684d91bf12b2698cf3c3b4" is_pre_election: true
I20260925 05:10:12.232157 4081 raft_consensus.cc:2454] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a5b29575aee4c2081b1b7483a6c6e9d in term 0.
I20260925 05:10:12.232241 3947 raft_consensus.cc:2454] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a5b29575aee4c2081b1b7483a6c6e9d in term 0.
I20260925 05:10:12.232383 3749 leader_election.cc:302] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5b29575aee4c2081b1b7483a6c6e9d, ba606fa79a284245b41276d5c312c023; no voters:
I20260925 05:10:12.232520 4152 raft_consensus.cc:2781] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:12.232579 4152 raft_consensus.cc:482] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:12.232609 4152 raft_consensus.cc:3037] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:12.233312 4152 raft_consensus.cc:504] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.233444 4152 leader_election.cc:288] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [CANDIDATE]: Term 1 election: Requested vote from peers 89f29459b0684d91bf12b2698cf3c3b4 (127.3.6.130:44745), ba606fa79a284245b41276d5c312c023 (127.3.6.131:34339)
I20260925 05:10:12.233603 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e8b47899634ff5bee5009056ac31c3" candidate_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f29459b0684d91bf12b2698cf3c3b4"
I20260925 05:10:12.233626 4081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e8b47899634ff5bee5009056ac31c3" candidate_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba606fa79a284245b41276d5c312c023"
I20260925 05:10:12.233690 3947 raft_consensus.cc:3037] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:12.233700 4081 raft_consensus.cc:3037] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:12.234594 3947 raft_consensus.cc:2454] T 52e8b47899634ff5bee5009056ac31c3 P 89f29459b0684d91bf12b2698cf3c3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a5b29575aee4c2081b1b7483a6c6e9d in term 1.
I20260925 05:10:12.234594 4081 raft_consensus.cc:2454] T 52e8b47899634ff5bee5009056ac31c3 P ba606fa79a284245b41276d5c312c023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a5b29575aee4c2081b1b7483a6c6e9d in term 1.
I20260925 05:10:12.234797 3749 leader_election.cc:302] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5b29575aee4c2081b1b7483a6c6e9d, ba606fa79a284245b41276d5c312c023; no voters:
I20260925 05:10:12.234897 4152 raft_consensus.cc:2781] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:12.235064 4152 raft_consensus.cc:686] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [term 1 LEADER]: Becoming Leader. State: Replica: 5a5b29575aee4c2081b1b7483a6c6e9d, State: Running, Role: LEADER
I20260925 05:10:12.235146 4152 consensus_queue.cc:237] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } }
I20260925 05:10:12.236156 3667 catalog_manager.cc:5495] T 52e8b47899634ff5bee5009056ac31c3 P 5a5b29575aee4c2081b1b7483a6c6e9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a5b29575aee4c2081b1b7483a6c6e9d (127.3.6.129). New cstate: current_term: 1 leader_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f29459b0684d91bf12b2698cf3c3b4" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba606fa79a284245b41276d5c312c023" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a5b29575aee4c2081b1b7483a6c6e9d" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46161 } health_report { overall_health: HEALTHY } } }
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:12.249734 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.242078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.3.6.129:46161)
0925 05:10:12.242250 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.242252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.242361 (+ 109us) server_negotiation.cc:407] Connection header received
0925 05:10:12.242403 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.242404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.242413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.242466 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:12.243016 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.243476 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.244288 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.244471 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.244522 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.246856 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:12.246865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:12.246865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:12.246871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:12.248598 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.248892 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.248895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.248895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.248979 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.249189 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.249190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.249191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.249336 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:12.249508 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.249545 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.249629 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:12.273633 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.266751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.3.6.129:46161)
0925 05:10:12.266857 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.266859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.266895 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:10:12.266956 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.266957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.266967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.267023 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:12.267591 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.268160 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.268916 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.269109 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.269162 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.271375 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:12.271377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:12.271377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:12.271384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:12.272653 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.272969 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.272970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.272970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.273021 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.273176 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.273176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.273176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.273266 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:12.273342 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.273482 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.273546 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:12.296717 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:12.289530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.3.6.129:46161)
0925 05:10:12.289626 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:12.289629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:12.289661 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:10:12.289760 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:12.289761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:12.289772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:12.289842 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:12.290420 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.290913 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.291669 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:12.291952 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:12.292026 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:12.294286 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:12.294287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:12.294288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:12.294295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:12.295505 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.295924 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.295925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.295926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.295987 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:12.296193 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:12.296194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:12.296194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:12.296298 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:12.296397 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:12.296507 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:10:12.296614 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 05:10:12.297156 3773 tablet_service.cc:2908] Rejecting scan request for tablet 52e8b47899634ff5bee5009056ac31c3: Uninitialized: clean time has not yet been initialized
I20260925 05:10:12.297945 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3730
I20260925 05:10:12.303455 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3864
I20260925 05:10:12.308272 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3998
I20260925 05:10:12.312829 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 3636
2026-09-25T05:10:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1975 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[4168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[4168](info): set up 2 sockets
Sep 25 05:10:12 dist-test-slave-1lvv krb5kdc[4168](info): commencing operation
krb5kdc: starting...
W20260925 05:10:13.363487 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:13 dist-test-slave-1lvv krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:13.490551 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:36973
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:13.575259 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:13.575557 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:13.575603 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:13.576895 4181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:13.576978 4181 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:10:13.577003 4181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:13.577029 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:13.577064 4181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:13.577091 4181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:13.578691 4181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:36973
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:13.579025 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:13.579283 4181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:10:13.584491 4181 server_base.cc:1034] running on GCE node
W20260925 05:10:13.584471 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:13.584470 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:13.584446 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:13.585080 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:13.585397 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:13.586542 4181 hybrid_clock.cc:648] HybridClock initialized: now 1790313013586544 us; error 46 us; skew 500 ppm
Sep 25 05:10:13 dist-test-slave-1lvv krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:13.593143 4181 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:13.593647 4181 webserver.cc:460] Webserver started at http://127.3.6.190:46305/ using document root <none> and password file <none>
I20260925 05:10:13.593887 4181 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:13.593962 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:13.594115 4181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:13.595214 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "8ea76c38ca91494e8a0fc4015e35068b"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "2234a266fd49f5c74d487cf5142c98ff"
server_key_iv: "871427f362019bdb1bb3fe5aa1a074a6"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.595588 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "8ea76c38ca91494e8a0fc4015e35068b"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "2234a266fd49f5c74d487cf5142c98ff"
server_key_iv: "871427f362019bdb1bb3fe5aa1a074a6"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.597015 4181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:13.597791 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:13.598007 4181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:13.598110 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "8ea76c38ca91494e8a0fc4015e35068b"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "2234a266fd49f5c74d487cf5142c98ff"
server_key_iv: "871427f362019bdb1bb3fe5aa1a074a6"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.598205 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:13.617654 4181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:13.620107 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:13.620298 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:13.625046 4181 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:36973
I20260925 05:10:13.625088 4258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:36973 every 8 connection(s)
I20260925 05:10:13.625463 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:13.626154 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 4181
I20260925 05:10:13.626186 4259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:13.626266 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:13.626470 3098 external_mini_cluster.cc:1428] Setting key 081e884cd763dfed676256df3e06b2d5
I20260925 05:10:13.629302 4259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: Bootstrap starting.
I20260925 05:10:13.629938 4259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:13.630226 4259 log.cc:824] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:13.631008 4259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: No bootstrap required, opened a new log
Sep 25 05:10:13 dist-test-slave-1lvv krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:13.633971 4259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea76c38ca91494e8a0fc4015e35068b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 36973 } }
I20260925 05:10:13.634146 4259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:13.634196 4259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea76c38ca91494e8a0fc4015e35068b, State: Initialized, Role: FOLLOWER
I20260925 05:10:13.634336 4259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea76c38ca91494e8a0fc4015e35068b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 36973 } }
I20260925 05:10:13.634433 4259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:13.634477 4259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:13.634523 4259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:13.635354 4259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea76c38ca91494e8a0fc4015e35068b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 36973 } }
I20260925 05:10:13.635471 4259 leader_election.cc:302] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ea76c38ca91494e8a0fc4015e35068b; no voters:
I20260925 05:10:13.635672 4259 leader_election.cc:288] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:13.635890 4265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:13.635998 4259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:13.636601 4265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [term 1 LEADER]: Becoming Leader. State: Replica: 8ea76c38ca91494e8a0fc4015e35068b, State: Running, Role: LEADER
I20260925 05:10:13.636735 4265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea76c38ca91494e8a0fc4015e35068b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 36973 } }
I20260925 05:10:13.637188 4265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ea76c38ca91494e8a0fc4015e35068b. Latest consensus state: current_term: 1 leader_uuid: "8ea76c38ca91494e8a0fc4015e35068b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea76c38ca91494e8a0fc4015e35068b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 36973 } } }
I20260925 05:10:13.637270 4265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [sys.catalog]: This master's current role is: LEADER
W20260925 05:10:13.638057 4278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:13.638118 4278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:10:13.638790 4264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ea76c38ca91494e8a0fc4015e35068b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea76c38ca91494e8a0fc4015e35068b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 36973 } } }
I20260925 05:10:13.638952 4264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:13.639595 4279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:13.639865 4279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:13.640264 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:13.627026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.3.6.190:36973)
0925 05:10:13.627241 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:13.627249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:13.627275 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:10:13.627725 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:13.627732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:13.627846 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:13.627995 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:13.628653 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.629639 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.630467 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.631086 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.631203 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:13.634950 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:13.634957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:13.634957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:13.634971 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:13.638126 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.638550 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.638553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.638553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.638671 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.638919 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.638920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.638921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.639256 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:13.639401 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:13.639884 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 05:10:13.640034 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260925 05:10:13.642002 4279 catalog_manager.cc:1347] Generated new cluster ID: 26cb558c26a84d068ebe1a7db343f72d
I20260925 05:10:13.642055 4279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:13.654605 4279 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:13.655165 4279 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:13.672356 4279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: Generated new TSK 0
I20260925 05:10:13.672592 4279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:13.706948 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:36973
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:13.791988 4285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:13.792207 4285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:13.792232 4285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:13.793516 4285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:13.793568 4285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:13.793589 4285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:13.793607 4285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:13.795167 4285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:36973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:13.795433 4285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:13.795679 4285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:13.800850 4293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:13.800876 4291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:13.800850 4290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:13.801165 4285 server_base.cc:1034] running on GCE node
I20260925 05:10:13.801383 4285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:13.801657 4285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:13.802822 4285 hybrid_clock.cc:648] HybridClock initialized: now 1790313013802812 us; error 42 us; skew 500 ppm
Sep 25 05:10:13 dist-test-slave-1lvv krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:13.805420 4285 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:13.805954 4285 webserver.cc:460] Webserver started at http://127.3.6.129:35667/ using document root <none> and password file <none>
I20260925 05:10:13.806164 4285 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:13.806205 4285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:13.806310 4285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:13.807376 4285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "64b92b3bd78f455d90987e5cbfdce127"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "d7ed497593a1cc15b2741806f591a4a2"
server_key_iv: "10f8b055602015ef19689c2386c1e9e5"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.807699 4285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "64b92b3bd78f455d90987e5cbfdce127"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "d7ed497593a1cc15b2741806f591a4a2"
server_key_iv: "10f8b055602015ef19689c2386c1e9e5"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.809167 4285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:13.809878 4300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:13.810032 4285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:13.810118 4285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "64b92b3bd78f455d90987e5cbfdce127"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "d7ed497593a1cc15b2741806f591a4a2"
server_key_iv: "10f8b055602015ef19689c2386c1e9e5"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.810176 4285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:13.822636 4285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:13.825045 4285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:13.825171 4285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:13.825665 4285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:13.825723 4285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:13.825750 4285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:13.825762 4285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:13.832582 4285 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:46461
I20260925 05:10:13.832669 4415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:46461 every 8 connection(s)
I20260925 05:10:13.832998 4285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:13 dist-test-slave-1lvv krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:13.836167 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:13.826208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:48697 (local address 127.3.6.190:36973)
0925 05:10:13.826280 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:13.826283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:13.826503 (+ 220us) server_negotiation.cc:407] Connection header received
0925 05:10:13.827022 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:13.827024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:13.827039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:13.827133 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:13.828112 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.828987 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.830571 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.830830 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.830921 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:13.833643 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:13.833644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:13.833645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:13.833651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:13.834785 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.835165 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.835165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.835165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.835219 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.835477 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.835478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.835479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.835653 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:13.835720 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:13.836019 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:10:13.836102 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:10:13.836352 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:13.826186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:36973 (local address 127.3.6.129:48697)
0925 05:10:13.826364 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:13.826407 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:13.826924 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:13.827202 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:13.827217 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:13.827624 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:13.827996 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:13.828003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.829107 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.829109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:13.830489 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:13.830492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.830934 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.830935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:13.831095 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:13.831747 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:13.831786 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:13.833535 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:13.834862 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:13.834868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:13.834870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:13.835095 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:13.835277 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:13.835277 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:13.835278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:13.835378 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:13.835715 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:13.835721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:13.835906 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 05:10:13.835996 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:10:13.836122 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260925 05:10:13.836930 4416 heartbeater.cc:347] Connected to a master server at 127.3.6.190:36973
I20260925 05:10:13.837013 4416 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:13.837213 4416 heartbeater.cc:511] Master 127.3.6.190:36973 requested a full tablet report, sending...
I20260925 05:10:13.837806 4213 ts_manager.cc:194] Registered new tserver with Master: 64b92b3bd78f455d90987e5cbfdce127 (127.3.6.129:46461)
I20260925 05:10:13.838616 4213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:48697
I20260925 05:10:13.842643 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 4285
I20260925 05:10:13.842756 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:13.842921 3098 external_mini_cluster.cc:1428] Setting key fdc7635fb98be63f985e322cdfbb8e88
I20260925 05:10:13.843411 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:13.838223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:49559 (local address 127.3.6.190:36973)
0925 05:10:13.838281 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:13.838284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:13.838407 (+ 123us) server_negotiation.cc:407] Connection header received
0925 05:10:13.838437 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:13.838438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:13.838447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:13.838509 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:13.838824 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.839356 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.840095 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.840272 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.840321 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:13.841123 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:13.841125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:13.841125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:13.841131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:13.842258 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.842573 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.842574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.842574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.842634 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:13.842861 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:13.842862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:13.842862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:13.842971 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:13.843053 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:13.843246 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 05:10:13.843322 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:10:13.843425 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:13.838156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:36973 (local address 127.3.6.129:49559)
0925 05:10:13.838338 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:13.838352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:13.838402 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:13.838512 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:13.838513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:13.838678 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:13.838751 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:13.838752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.839453 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.839454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:13.840021 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:13.840023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:13.840328 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:13.840329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:13.840381 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:13.840778 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:13.840782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:13.841052 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:13.842323 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:13.842324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:13.842325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:13.842489 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:13.842714 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:13.842715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:13.842715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:13.842777 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:13.843050 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:13.843052 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:13.843109 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 05:10:13.843201 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 05:10:13.843316 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:13.871295 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:36973
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:13.955189 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:13.955417 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:13.955442 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:13.956637 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:13.956676 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:13.956697 4425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:13.956715 4425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:13.958107 4425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:36973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:13.958303 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:13.958504 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:13.963730 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:13.963832 4425 server_base.cc:1034] running on GCE node
W20260925 05:10:13.963733 4433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:13.963730 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:13.964280 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:13.964557 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:13.965757 4425 hybrid_clock.cc:648] HybridClock initialized: now 1790313013965746 us; error 35 us; skew 500 ppm
Sep 25 05:10:13 dist-test-slave-1lvv krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:13.968637 4425 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:13.969255 4425 webserver.cc:460] Webserver started at http://127.3.6.130:36405/ using document root <none> and password file <none>
I20260925 05:10:13.969583 4425 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:13.969660 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:13.969817 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:13.970973 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "e392946b77974f68af0907d12daab87d"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "be6c6573a8315f2da6680c4cef5e5d6f"
server_key_iv: "65b01c3c9d451b5cdf5d1b3477f82d5c"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.971413 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "e392946b77974f68af0907d12daab87d"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "be6c6573a8315f2da6680c4cef5e5d6f"
server_key_iv: "65b01c3c9d451b5cdf5d1b3477f82d5c"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.973023 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 05:10:13.973822 4440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:13.973990 4425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:13.974093 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "e392946b77974f68af0907d12daab87d"
format_stamp: "Formatted at 2026-09-25 05:10:13 on dist-test-slave-1lvv"
server_key: "be6c6573a8315f2da6680c4cef5e5d6f"
server_key_iv: "65b01c3c9d451b5cdf5d1b3477f82d5c"
server_key_version: "encryptionkey@0"
I20260925 05:10:13.974170 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:14.010351 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:14.012966 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:14.013182 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:14.013747 4425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:14.013815 4425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.013863 4425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:14.013891 4425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:14.021588 4425 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:44479
I20260925 05:10:14.021673 4555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:44479 every 8 connection(s)
I20260925 05:10:14.021982 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:14.024694 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.014301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:49817 (local address 127.3.6.190:36973)
0925 05:10:14.014343 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.014345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.014638 (+ 293us) server_negotiation.cc:407] Connection header received
0925 05:10:14.015204 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.015206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.015221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.015280 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.016361 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.016932 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.018733 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.018948 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.019031 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.021515 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.021517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.021517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.021526 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.023200 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.023648 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.023649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.023650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.023715 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.023999 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.024000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.024001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.024135 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.024208 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.024496 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.024601 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:10:14.024893 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.014308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:36973 (local address 127.3.6.130:49817)
0925 05:10:14.014488 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.014523 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.015081 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.015412 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.015418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.015914 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:14.016257 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.016267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.017011 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.017012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.018586 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.018590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.019065 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.019067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.019233 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.019866 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:14.019875 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:14.021405 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:14.023313 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.023320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.023322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.023566 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.023806 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.023807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.023808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.023930 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.024209 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:14.024216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:14.024365 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.024473 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.024650 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260925 05:10:14.025406 4556 heartbeater.cc:347] Connected to a master server at 127.3.6.190:36973
I20260925 05:10:14.025521 4556 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:14.025717 4556 heartbeater.cc:511] Master 127.3.6.190:36973 requested a full tablet report, sending...
I20260925 05:10:14.026186 4212 ts_manager.cc:194] Registered new tserver with Master: e392946b77974f68af0907d12daab87d (127.3.6.130:44479)
I20260925 05:10:14.026790 4212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:49817
I20260925 05:10:14.027446 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 4425
I20260925 05:10:14.027545 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:14.027733 3098 external_mini_cluster.cc:1428] Setting key 94464f59821b75078c422666c5747745
I20260925 05:10:14.032716 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.026723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:38593 (local address 127.3.6.190:36973)
0925 05:10:14.026770 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.026773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.026841 (+ 68us) server_negotiation.cc:407] Connection header received
0925 05:10:14.026959 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.026960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.026971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.027001 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.027416 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.028130 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.028904 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.029115 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.029194 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.030328 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.030329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.030329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.030336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.031601 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.032018 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.032018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.032019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.032067 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.032256 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.032256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.032257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.032381 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.032429 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.032595 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.032654 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:10:14.032845 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.026588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:36973 (local address 127.3.6.130:38593)
0925 05:10:14.026773 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.026784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.026871 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.027063 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.027063 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.027274 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:14.027328 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.027330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.028225 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.028226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.028814 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.028816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.029196 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.029197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.029273 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.029768 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:14.029773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:14.030242 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:14.031676 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.031677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.031678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.031946 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.032127 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.032128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.032128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.032199 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.032442 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:14.032443 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:14.032493 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.032578 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.032703 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:14.055003 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:36973
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:14.138918 4565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:14.139168 4565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:14.139214 4565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:14.140686 4565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:14.140755 4565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:14.140792 4565 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:14.140844 4565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:14.142338 4565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:36973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:14.142616 4565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:14.142845 4565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:14.148003 4570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:14.148010 4571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:14.148092 4573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:14.148286 4565 server_base.cc:1034] running on GCE node
I20260925 05:10:14.148519 4565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:14.148756 4565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:14.149919 4565 hybrid_clock.cc:648] HybridClock initialized: now 1790313014149907 us; error 43 us; skew 500 ppm
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:14.152475 4565 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:14.152959 4565 webserver.cc:460] Webserver started at http://127.3.6.131:42967/ using document root <none> and password file <none>
I20260925 05:10:14.153261 4565 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:14.153333 4565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:14.153483 4565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:14.154588 4565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "b390af78234547f8b76d6b13e33bf190"
format_stamp: "Formatted at 2026-09-25 05:10:14 on dist-test-slave-1lvv"
server_key: "47e0e710ae271624407e3255eeeed4e7"
server_key_iv: "5b1d2e03a3443a812d43bc9b33af908a"
server_key_version: "encryptionkey@0"
I20260925 05:10:14.154963 4565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "b390af78234547f8b76d6b13e33bf190"
format_stamp: "Formatted at 2026-09-25 05:10:14 on dist-test-slave-1lvv"
server_key: "47e0e710ae271624407e3255eeeed4e7"
server_key_iv: "5b1d2e03a3443a812d43bc9b33af908a"
server_key_version: "encryptionkey@0"
I20260925 05:10:14.156387 4565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 05:10:14.157123 4580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.157311 4565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:14.157408 4565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "b390af78234547f8b76d6b13e33bf190"
format_stamp: "Formatted at 2026-09-25 05:10:14 on dist-test-slave-1lvv"
server_key: "47e0e710ae271624407e3255eeeed4e7"
server_key_iv: "5b1d2e03a3443a812d43bc9b33af908a"
server_key_version: "encryptionkey@0"
I20260925 05:10:14.157485 4565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:14.173863 4565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:14.176586 4565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:14.176712 4565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:14.177261 4565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:14.177309 4565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.177332 4565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:14.177364 4565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.183679 4565 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:37101
I20260925 05:10:14.183847 4695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:37101 every 8 connection(s)
I20260925 05:10:14.184170 4565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:14.187903 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.177750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:55751 (local address 127.3.6.190:36973)
0925 05:10:14.177820 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.177822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.178024 (+ 202us) server_negotiation.cc:407] Connection header received
0925 05:10:14.178628 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.178630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.178645 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.178733 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.179579 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.180456 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.182018 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.182345 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.182435 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.185182 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.185183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.185184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.185192 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.186441 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.186871 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.186872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.186872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.186938 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.187211 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.187212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.187213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.187338 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.187409 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.187695 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.187811 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:10:14.188109 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.177729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:36973 (local address 127.3.6.131:55751)
0925 05:10:14.177893 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.177933 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.178526 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.178797 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.178800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.179186 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:14.179474 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.179482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.180561 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.180562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.181921 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.181924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.182447 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.182449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.182631 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.183245 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:14.183254 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:14.185077 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:14.186552 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.186559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.186561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.186790 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.187019 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.187020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.187021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.187132 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.187411 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:14.187417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:14.187564 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.187674 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.187859 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260925 05:10:14.188735 4696 heartbeater.cc:347] Connected to a master server at 127.3.6.190:36973
I20260925 05:10:14.188832 4696 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:14.189036 4696 heartbeater.cc:511] Master 127.3.6.190:36973 requested a full tablet report, sending...
I20260925 05:10:14.189441 4213 ts_manager.cc:194] Registered new tserver with Master: b390af78234547f8b76d6b13e33bf190 (127.3.6.131:37101)
I20260925 05:10:14.189924 4213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:55751
I20260925 05:10:14.190604 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 4565
I20260925 05:10:14.190713 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:14.190871 3098 external_mini_cluster.cc:1428] Setting key 6dcacd3a840d3c0e6a54187fc4c4fecd
I20260925 05:10:14.191602 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:14.196267 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.189819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:36973 (local address 127.3.6.131:43393)
0925 05:10:14.189961 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.189974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.190061 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.190211 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.190212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.190391 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:14.190474 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.190477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.191348 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.191349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.192175 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.192178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.192553 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.192554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.192639 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.193076 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:14.193081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:14.193501 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:14.195315 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.195316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.195317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.195478 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.195677 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.195677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.195677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.195722 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.196012 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:14.196013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:14.196050 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.196106 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.196190 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 05:10:14.196390 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.189883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:43393 (local address 127.3.6.190:36973)
0925 05:10:14.189924 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.189927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.190035 (+ 108us) server_negotiation.cc:407] Connection header received
0925 05:10:14.190134 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.190135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.190147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.190205 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.190568 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.191251 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.192276 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.192469 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.192534 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.193608 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.193609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.193609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.193615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.195232 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.195548 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.195549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.195550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.195615 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.195802 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.195803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.195803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.195950 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.196025 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.196145 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.196245 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:14.217355 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.209777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.3.6.190:36973)
0925 05:10:14.209842 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.209844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.209857 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:10:14.209889 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.209890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.209901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.209968 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.210601 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.211102 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.211917 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.212112 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.212185 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.214614 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.214615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.214616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.214624 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.216201 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.216575 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.216576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.216577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.216632 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.216823 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.216824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.216825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.216943 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.217050 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.217164 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.217248 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:10:14.219113 4213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:14.220870 4213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:14.227602 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.223534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:46461 (local address 127.0.0.1:55602)
0925 05:10:14.223700 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.223711 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.223774 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.224256 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.224257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.224259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:14.224316 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.224318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.225561 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.225562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.226301 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.226303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.227247 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.227248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.227339 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.227355 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.227434 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.227502 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260925 05:10:14.227626 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.223815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.3.6.129:46461)
0925 05:10:14.224020 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.224025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.224035 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:14.224075 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.224077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.224093 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.224182 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:14.224406 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.225465 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.226391 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.227149 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.227245 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.227315 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.227472 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.227523 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260925 05:10:14.227722 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.223326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.3.6.131:37101)
0925 05:10:14.223545 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.223550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.223795 (+ 245us) server_negotiation.cc:407] Connection header received
0925 05:10:14.223924 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.223927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.223942 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.223994 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:14.224235 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.224981 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.226475 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.227229 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.227299 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.227348 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.227564 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.227610 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260925 05:10:14.227973 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.223233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37101 (local address 127.0.0.1:34038)
0925 05:10:14.223673 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.223701 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.223828 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.224083 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.224084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.224086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:14.224127 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.224128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.225071 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.225073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.226394 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.226397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.227383 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.227389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.227469 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.227472 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.227874 (+ 402us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.227922 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260925 05:10:14.228179 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.223233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:44479 (local address 127.0.0.1:60046)
0925 05:10:14.223423 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.223489 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.223556 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.223944 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.223947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.223950 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:14.224026 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.224028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.225415 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.225416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.226754 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.226756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.227588 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.227589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.227661 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.227666 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.228060 (+ 394us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.228106 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":32,"thread_start_us":76,"threads_started":1}
I20260925 05:10:14.228487 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.223432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.3.6.130:44479)
0925 05:10:14.223636 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.223640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.223651 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:14.223709 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.223712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.223730 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.223852 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:14.224111 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.225322 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.226831 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.227438 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.227500 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.227555 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.228323 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.228373 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260925 05:10:14.228489 4350 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda14849cda48649dba3de413623eb1 (DEFAULT_TABLE table=test-table [id=d9a35f198756461fb65506eafb2a1dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:14.228583 4630 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda14849cda48649dba3de413623eb1 (DEFAULT_TABLE table=test-table [id=d9a35f198756461fb65506eafb2a1dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:14.228890 4350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda14849cda48649dba3de413623eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.228890 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda14849cda48649dba3de413623eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.231707 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda14849cda48649dba3de413623eb1 (DEFAULT_TABLE table=test-table [id=d9a35f198756461fb65506eafb2a1dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:14.232091 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda14849cda48649dba3de413623eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.232337 4719 tablet_bootstrap.cc:492] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Bootstrap starting.
I20260925 05:10:14.232671 4720 tablet_bootstrap.cc:492] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Bootstrap starting.
I20260925 05:10:14.233232 4720 tablet_bootstrap.cc:654] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.233438 4719 tablet_bootstrap.cc:654] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.233587 4720 log.cc:824] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:14.233771 4719 log.cc:824] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:14.234460 4720 tablet_bootstrap.cc:492] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: No bootstrap required, opened a new log
I20260925 05:10:14.234494 4719 tablet_bootstrap.cc:492] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: No bootstrap required, opened a new log
I20260925 05:10:14.234544 4720 ts_tablet_manager.cc:1397] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:14.234650 4719 ts_tablet_manager.cc:1397] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:14.236155 4723 tablet_bootstrap.cc:492] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Bootstrap starting.
I20260925 05:10:14.236714 4719 raft_consensus.cc:348] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.236773 4720 raft_consensus.cc:348] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.236908 4719 raft_consensus.cc:374] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.236925 4720 raft_consensus.cc:374] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.236965 4723 tablet_bootstrap.cc:654] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.236963 4720 raft_consensus.cc:729] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b92b3bd78f455d90987e5cbfdce127, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.236961 4719 raft_consensus.cc:729] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b390af78234547f8b76d6b13e33bf190, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.237085 4720 consensus_queue.cc:260] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.237112 4719 consensus_queue.cc:260] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.237370 4720 ts_tablet_manager.cc:1428] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:14.237363 4723 log.cc:824] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:14.237425 4719 ts_tablet_manager.cc:1428] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:10:14.237504 4416 heartbeater.cc:503] Master 127.3.6.190:36973 was elected leader, sending a full tablet report...
I20260925 05:10:14.237538 4696 heartbeater.cc:503] Master 127.3.6.190:36973 was elected leader, sending a full tablet report...
I20260925 05:10:14.238828 4723 tablet_bootstrap.cc:492] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: No bootstrap required, opened a new log
I20260925 05:10:14.238900 4723 ts_tablet_manager.cc:1397] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:10:14.240434 4723 raft_consensus.cc:348] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.240566 4723 raft_consensus.cc:374] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.240587 4723 raft_consensus.cc:729] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e392946b77974f68af0907d12daab87d, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.240667 4723 consensus_queue.cc:260] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.240885 4723 ts_tablet_manager.cc:1428] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:14.240931 4556 heartbeater.cc:503] Master 127.3.6.190:36973 was elected leader, sending a full tablet report...
W20260925 05:10:14.272686 4557 tablet.cc:2369] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:14.307196 4724 raft_consensus.cc:482] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:14.307343 4724 raft_consensus.cc:504] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.307749 4724 leader_election.cc:288] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b390af78234547f8b76d6b13e33bf190 (127.3.6.131:37101), e392946b77974f68af0907d12daab87d (127.3.6.130:44479)
I20260925 05:10:14.310858 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.308105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:47661 (local address 127.3.6.131:37101)
0925 05:10:14.308156 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.308158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.308280 (+ 122us) server_negotiation.cc:407] Connection header received
0925 05:10:14.308346 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.308347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.308359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.308428 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:14.308578 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.309035 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.310130 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.310491 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.310574 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.310586 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.310739 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.310779 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:10:14.310883 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.308082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:44595 (local address 127.3.6.130:44479)
0925 05:10:14.308137 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.308140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.308158 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:10:14.308269 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.308270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.308306 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.308361 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:14.308528 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.309015 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.310138 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.310499 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.310580 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.310591 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.310774 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.310815 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:10:14.310886 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.308010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:44479 (local address 127.3.6.129:44595)
0925 05:10:14.308066 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.308113 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.308170 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.308369 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.308370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.308373 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:14.308451 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.308453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.309108 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.309109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.310051 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.310054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.310580 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.310581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.310656 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.310659 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.310738 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.310793 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260925 05:10:14.310899 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.308011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37101 (local address 127.3.6.129:47661)
0925 05:10:14.308226 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.308236 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.308290 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.308431 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.308432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.308434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:14.308515 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.308517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.309123 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.309124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.310030 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.310033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.310582 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.310583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.310656 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.310660 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.310721 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.310764 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":24,"thread_start_us":136,"threads_started":1}
I20260925 05:10:14.311307 4510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda14849cda48649dba3de413623eb1" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e392946b77974f68af0907d12daab87d" is_pre_election: true
I20260925 05:10:14.311406 4650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda14849cda48649dba3de413623eb1" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b390af78234547f8b76d6b13e33bf190" is_pre_election: true
I20260925 05:10:14.311494 4510 raft_consensus.cc:2454] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 0.
I20260925 05:10:14.311535 4650 raft_consensus.cc:2454] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 0.
I20260925 05:10:14.311810 4304 leader_election.cc:302] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b92b3bd78f455d90987e5cbfdce127, e392946b77974f68af0907d12daab87d; no voters:
I20260925 05:10:14.311941 4724 raft_consensus.cc:2781] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:14.311998 4724 raft_consensus.cc:482] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:14.312022 4724 raft_consensus.cc:3037] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.312868 4724 raft_consensus.cc:504] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.313011 4724 leader_election.cc:288] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 election: Requested vote from peers b390af78234547f8b76d6b13e33bf190 (127.3.6.131:37101), e392946b77974f68af0907d12daab87d (127.3.6.130:44479)
I20260925 05:10:14.313207 4510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda14849cda48649dba3de413623eb1" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e392946b77974f68af0907d12daab87d"
I20260925 05:10:14.313241 4650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda14849cda48649dba3de413623eb1" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b390af78234547f8b76d6b13e33bf190"
I20260925 05:10:14.313321 4650 raft_consensus.cc:3037] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.313329 4510 raft_consensus.cc:3037] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.314148 4650 raft_consensus.cc:2454] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 1.
I20260925 05:10:14.314271 4510 raft_consensus.cc:2454] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 1.
I20260925 05:10:14.314646 4302 leader_election.cc:302] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b92b3bd78f455d90987e5cbfdce127, b390af78234547f8b76d6b13e33bf190; no voters:
I20260925 05:10:14.314755 4724 raft_consensus.cc:2781] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:14.314869 4724 raft_consensus.cc:686] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 LEADER]: Becoming Leader. State: Replica: 64b92b3bd78f455d90987e5cbfdce127, State: Running, Role: LEADER
I20260925 05:10:14.314966 4724 consensus_queue.cc:237] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } }
I20260925 05:10:14.315910 4213 catalog_manager.cc:5495] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64b92b3bd78f455d90987e5cbfdce127 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } health_report { overall_health: UNKNOWN } } }
W20260925 05:10:14.333858 4417 tablet.cc:2369] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:14.343542 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.341124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.3.6.129:46461)
0925 05:10:14.341182 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.341184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.341245 (+ 61us) server_negotiation.cc:407] Connection header received
0925 05:10:14.341330 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.341331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.341350 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.341414 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:14.341564 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.342029 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.342799 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.342984 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.343038 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.343215 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.343305 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.343364 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.343403 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.343422 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:10:14.345386 4510 raft_consensus.cc:1261] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Refusing update from remote peer 64b92b3bd78f455d90987e5cbfdce127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:14.345419 4650 raft_consensus.cc:1261] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Refusing update from remote peer 64b92b3bd78f455d90987e5cbfdce127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:14.345746 4724 consensus_queue.cc:1035] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:14.345899 4729 consensus_queue.cc:1035] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:14.348153 4732 mvcc.cc:205] Tried to move back new op lower bound from 7333122106755977216 to 7333122106634862592. Current Snapshot: MvccSnapshot[applied={T|T < 7333122106755977216}]
I20260925 05:10:14.348405 4733 mvcc.cc:205] Tried to move back new op lower bound from 7333122106755977216 to 7333122106634862592. Current Snapshot: MvccSnapshot[applied={T|T < 7333122106755977216}]
I20260925 05:10:14.348853 4734 mvcc.cc:205] Tried to move back new op lower bound from 7333122106755977216 to 7333122106634862592. Current Snapshot: MvccSnapshot[applied={T|T < 7333122106755977216}]
I20260925 05:10:14.352329 4213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:14.352427 4213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:14.353214 4213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: Sending DeleteTablet for 3 replicas of tablet 6dda14849cda48649dba3de413623eb1
I20260925 05:10:14.353653 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dda14849cda48649dba3de413623eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:60046
I20260925 05:10:14.353667 4350 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dda14849cda48649dba3de413623eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:55602
I20260925 05:10:14.353897 4630 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dda14849cda48649dba3de413623eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:34038
I20260925 05:10:14.354126 4744 tablet_replica.cc:330] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: stopping tablet replica
I20260925 05:10:14.354197 4742 tablet_replica.cc:330] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: stopping tablet replica
I20260925 05:10:14.354295 4742 raft_consensus.cc:2229] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:14.354267 4213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:10:14.354417 4742 raft_consensus.cc:2258] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 05:10:14.354435 4213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:14.354657 4743 tablet_replica.cc:330] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: stopping tablet replica
I20260925 05:10:14.354929 4742 ts_tablet_manager.cc:1905] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:14.355742 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 51d8cdfd9048409ca4643fd55b3b17c9 (DEFAULT_TABLE table=test-table [id=4de230ca6ab24386a6dfa9b51d5d571a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:14.355901 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51d8cdfd9048409ca4643fd55b3b17c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.356055 4630 tablet_service.cc:1467] Processing CreateTablet for tablet 51d8cdfd9048409ca4643fd55b3b17c9 (DEFAULT_TABLE table=test-table [id=4de230ca6ab24386a6dfa9b51d5d571a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:14.356158 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51d8cdfd9048409ca4643fd55b3b17c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.357023 4742 ts_tablet_manager.cc:1918] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:14.357092 4742 log.cc:1196] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/6dda14849cda48649dba3de413623eb1
I20260925 05:10:14.357364 4742 ts_tablet_manager.cc:1939] T 6dda14849cda48649dba3de413623eb1 P e392946b77974f68af0907d12daab87d: Deleting consensus metadata
I20260925 05:10:14.357573 4719 tablet_bootstrap.cc:492] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Bootstrap starting.
I20260925 05:10:14.357594 4723 tablet_bootstrap.cc:492] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: Bootstrap starting.
I20260925 05:10:14.357776 4200 catalog_manager.cc:4841] TS e392946b77974f68af0907d12daab87d (127.3.6.130:44479): tablet 6dda14849cda48649dba3de413623eb1 (table test-table [id=d9a35f198756461fb65506eafb2a1dec]) successfully deleted
I20260925 05:10:14.358011 4719 tablet_bootstrap.cc:654] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.358011 4723 tablet_bootstrap.cc:654] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.358605 4744 raft_consensus.cc:2229] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:14.358762 4744 raft_consensus.cc:2258] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:14.358505 4350 tablet_service.cc:1467] Processing CreateTablet for tablet 51d8cdfd9048409ca4643fd55b3b17c9 (DEFAULT_TABLE table=test-table [id=4de230ca6ab24386a6dfa9b51d5d571a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:14.358951 4719 tablet_bootstrap.cc:492] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: No bootstrap required, opened a new log
I20260925 05:10:14.358991 4350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51d8cdfd9048409ca4643fd55b3b17c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.359014 4719 ts_tablet_manager.cc:1397] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:14.359163 4719 raft_consensus.cc:348] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.359252 4719 raft_consensus.cc:374] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.359275 4719 raft_consensus.cc:729] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b390af78234547f8b76d6b13e33bf190, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.359323 4719 consensus_queue.cc:260] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.359411 4719 ts_tablet_manager.cc:1428] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:14.359805 4723 tablet_bootstrap.cc:492] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: No bootstrap required, opened a new log
I20260925 05:10:14.359855 4723 ts_tablet_manager.cc:1397] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:14.359997 4723 raft_consensus.cc:348] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.360072 4723 raft_consensus.cc:374] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.360106 4723 raft_consensus.cc:729] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e392946b77974f68af0907d12daab87d, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.360172 4723 consensus_queue.cc:260] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.360265 4723 ts_tablet_manager.cc:1428] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.360462 4720 tablet_bootstrap.cc:492] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: Bootstrap starting.
I20260925 05:10:14.354765 4743 raft_consensus.cc:2229] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:14.360905 4720 tablet_bootstrap.cc:654] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.360986 4743 raft_consensus.cc:2258] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:14.361465 4744 ts_tablet_manager.cc:1905] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:14.361606 4720 tablet_bootstrap.cc:492] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: No bootstrap required, opened a new log
I20260925 05:10:14.361657 4720 ts_tablet_manager.cc:1397] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:14.361804 4720 raft_consensus.cc:348] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.361881 4720 raft_consensus.cc:374] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.361902 4720 raft_consensus.cc:729] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b92b3bd78f455d90987e5cbfdce127, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.361948 4720 consensus_queue.cc:260] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.362042 4720 ts_tablet_manager.cc:1428] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:14.363299 4743 ts_tablet_manager.cc:1905] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:14.364485 4744 ts_tablet_manager.cc:1918] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:14.364578 4744 log.cc:1196] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/6dda14849cda48649dba3de413623eb1
I20260925 05:10:14.364562 4743 ts_tablet_manager.cc:1918] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:14.364639 4743 log.cc:1196] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/6dda14849cda48649dba3de413623eb1
I20260925 05:10:14.364846 4744 ts_tablet_manager.cc:1939] T 6dda14849cda48649dba3de413623eb1 P b390af78234547f8b76d6b13e33bf190: Deleting consensus metadata
I20260925 05:10:14.364919 4743 ts_tablet_manager.cc:1939] T 6dda14849cda48649dba3de413623eb1 P 64b92b3bd78f455d90987e5cbfdce127: Deleting consensus metadata
I20260925 05:10:14.365280 4198 catalog_manager.cc:4841] TS 64b92b3bd78f455d90987e5cbfdce127 (127.3.6.129:46461): tablet 6dda14849cda48649dba3de413623eb1 (table test-table [id=d9a35f198756461fb65506eafb2a1dec]) successfully deleted
I20260925 05:10:14.365391 4198 catalog_manager.cc:4841] TS b390af78234547f8b76d6b13e33bf190 (127.3.6.131:37101): tablet 6dda14849cda48649dba3de413623eb1 (table test-table [id=d9a35f198756461fb65506eafb2a1dec]) successfully deleted
I20260925 05:10:14.391971 4725 raft_consensus.cc:482] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:14.392092 4725 raft_consensus.cc:504] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.392457 4725 leader_election.cc:288] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b92b3bd78f455d90987e5cbfdce127 (127.3.6.129:46461), e392946b77974f68af0907d12daab87d (127.3.6.130:44479)
I20260925 05:10:14.396041 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.392863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:56929 (local address 127.3.6.129:46461)
0925 05:10:14.392906 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.392908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.392920 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:14.392958 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.392959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.392969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.393029 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:14.393243 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.393811 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.395059 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.395652 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.395728 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.395740 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.395923 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.395966 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:10:14.396067 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.392691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:46461 (local address 127.3.6.131:56929)
0925 05:10:14.392829 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.392840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.392882 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.393087 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.393088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.393090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:14.393164 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.393167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.393894 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.393895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.394952 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.394955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.395738 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.395739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.395846 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.395850 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.395907 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.395953 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260925 05:10:14.396135 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.392780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:38139 (local address 127.3.6.130:44479)
0925 05:10:14.392826 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.392829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.393097 (+ 268us) server_negotiation.cc:407] Connection header received
0925 05:10:14.393213 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.393214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.393225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.393259 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:14.393463 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.394201 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.395353 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.395729 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.395826 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.395838 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.396018 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.396061 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:14.396543 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.392701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:44479 (local address 127.3.6.131:38139)
0925 05:10:14.393029 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.393041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.393121 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.393322 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.393323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.393324 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:14.393386 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.393387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.394296 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.394296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.395272 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.395274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.395869 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.395870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.395937 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.395940 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.396425 (+ 485us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.396466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260925 05:10:14.396453 4370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d8cdfd9048409ca4643fd55b3b17c9" candidate_uuid: "b390af78234547f8b76d6b13e33bf190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b92b3bd78f455d90987e5cbfdce127" is_pre_election: true
I20260925 05:10:14.396611 4370 raft_consensus.cc:2454] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b390af78234547f8b76d6b13e33bf190 in term 0.
I20260925 05:10:14.396842 4582 leader_election.cc:302] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b92b3bd78f455d90987e5cbfdce127, b390af78234547f8b76d6b13e33bf190; no voters:
I20260925 05:10:14.396905 4510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d8cdfd9048409ca4643fd55b3b17c9" candidate_uuid: "b390af78234547f8b76d6b13e33bf190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e392946b77974f68af0907d12daab87d" is_pre_election: true
I20260925 05:10:14.396992 4510 raft_consensus.cc:2454] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b390af78234547f8b76d6b13e33bf190 in term 0.
I20260925 05:10:14.397002 4725 raft_consensus.cc:2781] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:14.397033 4725 raft_consensus.cc:482] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:14.397056 4725 raft_consensus.cc:3037] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.397787 4725 raft_consensus.cc:504] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.397938 4725 leader_election.cc:288] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [CANDIDATE]: Term 1 election: Requested vote from peers 64b92b3bd78f455d90987e5cbfdce127 (127.3.6.129:46461), e392946b77974f68af0907d12daab87d (127.3.6.130:44479)
I20260925 05:10:14.398092 4510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d8cdfd9048409ca4643fd55b3b17c9" candidate_uuid: "b390af78234547f8b76d6b13e33bf190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e392946b77974f68af0907d12daab87d"
I20260925 05:10:14.398175 4510 raft_consensus.cc:3037] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.398198 4370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d8cdfd9048409ca4643fd55b3b17c9" candidate_uuid: "b390af78234547f8b76d6b13e33bf190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b92b3bd78f455d90987e5cbfdce127"
I20260925 05:10:14.398301 4370 raft_consensus.cc:3037] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.398813 4510 raft_consensus.cc:2454] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b390af78234547f8b76d6b13e33bf190 in term 1.
I20260925 05:10:14.398988 4584 leader_election.cc:302] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b390af78234547f8b76d6b13e33bf190, e392946b77974f68af0907d12daab87d; no voters:
I20260925 05:10:14.399125 4725 raft_consensus.cc:2781] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:14.399240 4370 raft_consensus.cc:2454] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b390af78234547f8b76d6b13e33bf190 in term 1.
I20260925 05:10:14.399279 4725 raft_consensus.cc:686] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 1 LEADER]: Becoming Leader. State: Replica: b390af78234547f8b76d6b13e33bf190, State: Running, Role: LEADER
I20260925 05:10:14.399360 4725 consensus_queue.cc:237] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.400117 4212 catalog_manager.cc:5495] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 reported cstate change: term changed from 0 to 1, leader changed from <none> to b390af78234547f8b76d6b13e33bf190 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "b390af78234547f8b76d6b13e33bf190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } health_report { overall_health: HEALTHY } } }
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:14.428133 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.420635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.3.6.190:36973)
0925 05:10:14.420692 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.420694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.420824 (+ 130us) server_negotiation.cc:407] Connection header received
0925 05:10:14.420898 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.420899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.420910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.420976 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.421450 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.421940 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.422632 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.422815 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.422896 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.425385 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.425387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.425387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.425395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.426870 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.427268 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.427269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.427270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.427326 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.427509 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.427510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.427510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.427628 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.427719 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.427917 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.428020 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:10:14.428229 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.420584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:36973 (local address 127.0.0.1:36938)
0925 05:10:14.420741 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.420756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.420845 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.420980 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.420981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.421307 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:14.421354 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.421356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.422023 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.422024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.422548 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.422549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.422897 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.422897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.422947 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.423547 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:14.423552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:14.425290 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:14.426955 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.426965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.426967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.427164 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.427388 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:14.427389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:14.427389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:14.427448 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:14.427709 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:14.427718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:14.427797 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.427892 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.428084 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":80,"thread_start_us":85,"threads_started":1}
I20260925 05:10:14.430830 4212 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:36938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:10:14.431039 4212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:14.432601 4350 tablet_service.cc:1467] Processing CreateTablet for tablet 10ad1dacacf7461d91d3deafe1159c0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74ed85daeda24c23b5b32714fe8817b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:14.432662 4490 tablet_service.cc:1467] Processing CreateTablet for tablet 10ad1dacacf7461d91d3deafe1159c0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74ed85daeda24c23b5b32714fe8817b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:14.432668 4630 tablet_service.cc:1467] Processing CreateTablet for tablet 10ad1dacacf7461d91d3deafe1159c0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74ed85daeda24c23b5b32714fe8817b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:14.432796 4350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ad1dacacf7461d91d3deafe1159c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.432811 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ad1dacacf7461d91d3deafe1159c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.432823 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ad1dacacf7461d91d3deafe1159c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:14.434441 4720 tablet_bootstrap.cc:492] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: Bootstrap starting.
I20260925 05:10:14.434463 4719 tablet_bootstrap.cc:492] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: Bootstrap starting.
W20260925 05:10:14.434770 4697 tablet.cc:2369] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:14.435016 4719 tablet_bootstrap.cc:654] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.435025 4720 tablet_bootstrap.cc:654] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.435407 4723 tablet_bootstrap.cc:492] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: Bootstrap starting.
I20260925 05:10:14.435710 4720 tablet_bootstrap.cc:492] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: No bootstrap required, opened a new log
I20260925 05:10:14.435746 4720 ts_tablet_manager.cc:1397] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:14.435938 4723 tablet_bootstrap.cc:654] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:14.436010 4720 raft_consensus.cc:348] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.436110 4720 raft_consensus.cc:374] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.436129 4720 raft_consensus.cc:729] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b92b3bd78f455d90987e5cbfdce127, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.436187 4720 consensus_queue.cc:260] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.436304 4720 ts_tablet_manager.cc:1428] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.436455 4737 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.436546 4737 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:14.437014 4723 tablet_bootstrap.cc:492] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: No bootstrap required, opened a new log
I20260925 05:10:14.437067 4723 ts_tablet_manager.cc:1397] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:14.437223 4723 raft_consensus.cc:348] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.437291 4723 raft_consensus.cc:374] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.437311 4723 raft_consensus.cc:729] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e392946b77974f68af0907d12daab87d, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.437358 4723 consensus_queue.cc:260] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.437465 4723 ts_tablet_manager.cc:1428] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.437512 4727 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.437619 4727 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:14.438356 4719 tablet_bootstrap.cc:492] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: No bootstrap required, opened a new log
I20260925 05:10:14.438405 4719 ts_tablet_manager.cc:1397] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 05:10:14.438654 4719 raft_consensus.cc:348] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.438719 4719 raft_consensus.cc:374] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:14.438733 4719 raft_consensus.cc:729] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b390af78234547f8b76d6b13e33bf190, State: Initialized, Role: FOLLOWER
I20260925 05:10:14.438792 4719 consensus_queue.cc:260] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.438907 4719 ts_tablet_manager.cc:1428] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:14.438934 4725 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.439004 4725 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:14.661916 4737 raft_consensus.cc:482] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:14.662065 4737 raft_consensus.cc:504] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.662281 4737 leader_election.cc:288] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e392946b77974f68af0907d12daab87d (127.3.6.130:44479), b390af78234547f8b76d6b13e33bf190 (127.3.6.131:37101)
I20260925 05:10:14.662473 4510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad1dacacf7461d91d3deafe1159c0c" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e392946b77974f68af0907d12daab87d" is_pre_election: true
I20260925 05:10:14.662559 4510 raft_consensus.cc:2454] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 0.
I20260925 05:10:14.662493 4650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad1dacacf7461d91d3deafe1159c0c" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b390af78234547f8b76d6b13e33bf190" is_pre_election: true
I20260925 05:10:14.662632 4650 raft_consensus.cc:2454] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 0.
I20260925 05:10:14.662740 4304 leader_election.cc:302] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b92b3bd78f455d90987e5cbfdce127, e392946b77974f68af0907d12daab87d; no voters:
I20260925 05:10:14.662850 4737 raft_consensus.cc:2781] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:14.662905 4737 raft_consensus.cc:482] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:14.662921 4737 raft_consensus.cc:3037] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.663728 4737 raft_consensus.cc:504] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.663895 4737 leader_election.cc:288] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 election: Requested vote from peers e392946b77974f68af0907d12daab87d (127.3.6.130:44479), b390af78234547f8b76d6b13e33bf190 (127.3.6.131:37101)
I20260925 05:10:14.664059 4650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad1dacacf7461d91d3deafe1159c0c" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b390af78234547f8b76d6b13e33bf190"
I20260925 05:10:14.664084 4510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad1dacacf7461d91d3deafe1159c0c" candidate_uuid: "64b92b3bd78f455d90987e5cbfdce127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e392946b77974f68af0907d12daab87d"
I20260925 05:10:14.664150 4510 raft_consensus.cc:3037] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.664153 4650 raft_consensus.cc:3037] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:14.664758 4650 raft_consensus.cc:2454] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 1.
I20260925 05:10:14.664840 4510 raft_consensus.cc:2454] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b92b3bd78f455d90987e5cbfdce127 in term 1.
I20260925 05:10:14.664942 4302 leader_election.cc:302] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b92b3bd78f455d90987e5cbfdce127, b390af78234547f8b76d6b13e33bf190; no voters:
I20260925 05:10:14.665055 4737 raft_consensus.cc:2781] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:14.665138 4737 raft_consensus.cc:686] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [term 1 LEADER]: Becoming Leader. State: Replica: 64b92b3bd78f455d90987e5cbfdce127, State: Running, Role: LEADER
I20260925 05:10:14.665207 4737 consensus_queue.cc:237] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } }
I20260925 05:10:14.665524 4729 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: TxnStatusTablet state changed. Reason: New leader 64b92b3bd78f455d90987e5cbfdce127. Latest consensus state: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.665621 4729 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:14.665853 4212 catalog_manager.cc:5495] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64b92b3bd78f455d90987e5cbfdce127 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:14.666242 4761 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:10:14.666388 4761 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:10:14.727811 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.725215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.3.6.129:46461)
0925 05:10:14.725261 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.725263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.725329 (+ 66us) server_negotiation.cc:407] Connection header received
0925 05:10:14.725408 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.725408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.725419 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.725493 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:14.725710 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.726196 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.727010 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.727219 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.727302 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.727481 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.727524 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.727598 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.727667 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.727692 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:14.727811 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.725154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:46461 (local address 127.0.0.1:55614)
0925 05:10:14.725273 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.725285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.725343 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.725508 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.725509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.725510 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:14.725581 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.725583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.726293 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.726294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.726914 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.726917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.727302 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.727303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.727373 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.727396 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.727606 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.727618 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.727665 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.727720 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 05:10:14.728943 4650 raft_consensus.cc:1261] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Refusing update from remote peer 64b92b3bd78f455d90987e5cbfdce127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:14.729048 4510 raft_consensus.cc:1261] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Refusing update from remote peer 64b92b3bd78f455d90987e5cbfdce127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:14.729225 4729 consensus_queue.cc:1035] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:14.729347 4737 consensus_queue.cc:1035] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:14.729912 4727 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: TxnStatusTablet state changed. Reason: New leader 64b92b3bd78f455d90987e5cbfdce127. Latest consensus state: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.730015 4727 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:14.730098 4725 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: TxnStatusTablet state changed. Reason: New leader 64b92b3bd78f455d90987e5cbfdce127. Latest consensus state: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.730185 4725 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:14.730723 4737 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.730834 4737 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:14.731050 4727 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.731139 4727 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P e392946b77974f68af0907d12daab87d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:14.732381 4724 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.732486 4724 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P 64b92b3bd78f455d90987e5cbfdce127: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:14.732779 4725 tablet_replica.cc:439] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64b92b3bd78f455d90987e5cbfdce127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 } } peers { permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 } } peers { permanent_uuid: "b390af78234547f8b76d6b13e33bf190" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37101 } } }
I20260925 05:10:14.732877 4725 tablet_replica.cc:442] T 10ad1dacacf7461d91d3deafe1159c0c P b390af78234547f8b76d6b13e33bf190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:14.735953 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.733081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.3.6.131:37101)
0925 05:10:14.733146 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.733148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.733158 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:14.733196 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.733197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.733218 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.733301 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:14.733454 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.733939 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.735017 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.735220 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.735270 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.735484 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.735614 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.735727 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.735827 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.735854 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:10:14.741206 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.738857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:36927 (local address 127.3.6.129:46461)
0925 05:10:14.738906 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.738908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.738921 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:10:14.738991 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.738992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.739005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.739066 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:14.739179 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.739650 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.740478 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.740681 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.740763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.740945 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.740984 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.741042 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.741101 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.741124 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:10:14.741221 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.738707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:46461 (local address 127.3.6.131:36927)
0925 05:10:14.738865 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.738877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.738937 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.739063 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.739064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.739066 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:14.739129 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.739130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.739728 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.739730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.740377 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.740379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.740764 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.740765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.740836 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.740850 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.741043 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.741046 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.741088 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.741133 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260925 05:10:14.743525 4509 raft_consensus.cc:1261] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Refusing update from remote peer b390af78234547f8b76d6b13e33bf190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:14.743616 4370 raft_consensus.cc:1261] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Refusing update from remote peer b390af78234547f8b76d6b13e33bf190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:14.743825 4749 consensus_queue.cc:1035] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e392946b77974f68af0907d12daab87d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:14.744155 4749 consensus_queue.cc:1035] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b92b3bd78f455d90987e5cbfdce127" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:14.758616 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.752279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56531 (local address 127.3.6.131:37101)
0925 05:10:14.752322 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.752324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.756029 (+ 3705us) server_negotiation.cc:407] Connection header received
0925 05:10:14.756099 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.756100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.756114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.756164 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:14.756318 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.756792 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.757921 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.758111 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.758167 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.758370 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.758397 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.758434 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.758519 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.758535 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:14.758662 4773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.752220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37101 (local address 127.3.6.129:56531)
0925 05:10:14.755936 (+ 3716us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:14.755950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:14.756036 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:14.756189 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:14.756190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:14.756192 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:14.756254 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.756256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.756881 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.756882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.757834 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:14.757838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.758196 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.758197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:14.758263 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.758285 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:14.758458 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:14.758460 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:14.758517 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:10:14.758571 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3658,"thread_start_us":105,"threads_started":1}
I20260925 05:10:14.771560 4212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:14.771706 4212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:14.772498 4212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ea76c38ca91494e8a0fc4015e35068b: Sending DeleteTablet for 3 replicas of tablet 51d8cdfd9048409ca4643fd55b3b17c9
I20260925 05:10:14.772886 4630 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d8cdfd9048409ca4643fd55b3b17c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:34038
I20260925 05:10:14.772951 4490 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d8cdfd9048409ca4643fd55b3b17c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:60046
I20260925 05:10:14.773007 4744 tablet_replica.cc:330] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: stopping tablet replica
I20260925 05:10:14.773064 4742 tablet_replica.cc:330] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: stopping tablet replica
I20260925 05:10:14.773066 4744 raft_consensus.cc:2229] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:14.773113 4742 raft_consensus.cc:2229] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:14.773154 4742 raft_consensus.cc:2258] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:14.773159 4744 raft_consensus.cc:2258] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:14.773412 4350 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d8cdfd9048409ca4643fd55b3b17c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:55602
I20260925 05:10:14.773514 4743 tablet_replica.cc:330] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: stopping tablet replica
I20260925 05:10:14.773561 4743 raft_consensus.cc:2229] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:14.773535 4744 ts_tablet_manager.cc:1905] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:14.773597 4743 raft_consensus.cc:2258] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:14.773984 4743 ts_tablet_manager.cc:1905] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:14.775594 4743 ts_tablet_manager.cc:1918] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:14.775650 4743 log.cc:1196] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/51d8cdfd9048409ca4643fd55b3b17c9
I20260925 05:10:14.775933 4743 ts_tablet_manager.cc:1939] T 51d8cdfd9048409ca4643fd55b3b17c9 P 64b92b3bd78f455d90987e5cbfdce127: Deleting consensus metadata
I20260925 05:10:14.776193 4744 ts_tablet_manager.cc:1918] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:14.776254 4744 log.cc:1196] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/51d8cdfd9048409ca4643fd55b3b17c9
I20260925 05:10:14.776352 4198 catalog_manager.cc:4841] TS 64b92b3bd78f455d90987e5cbfdce127 (127.3.6.129:46461): tablet 51d8cdfd9048409ca4643fd55b3b17c9 (table test-table [id=4de230ca6ab24386a6dfa9b51d5d571a]) successfully deleted
I20260925 05:10:14.776479 4744 ts_tablet_manager.cc:1939] T 51d8cdfd9048409ca4643fd55b3b17c9 P b390af78234547f8b76d6b13e33bf190: Deleting consensus metadata
I20260925 05:10:14.776664 4742 ts_tablet_manager.cc:1905] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:14.776813 4198 catalog_manager.cc:4841] TS b390af78234547f8b76d6b13e33bf190 (127.3.6.131:37101): tablet 51d8cdfd9048409ca4643fd55b3b17c9 (table test-table [id=4de230ca6ab24386a6dfa9b51d5d571a]) successfully deleted
I20260925 05:10:14.778294 4742 ts_tablet_manager.cc:1918] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:14.778363 4742 log.cc:1196] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/51d8cdfd9048409ca4643fd55b3b17c9
I20260925 05:10:14.778576 4742 ts_tablet_manager.cc:1939] T 51d8cdfd9048409ca4643fd55b3b17c9 P e392946b77974f68af0907d12daab87d: Deleting consensus metadata
I20260925 05:10:14.778937 4200 catalog_manager.cc:4841] TS e392946b77974f68af0907d12daab87d (127.3.6.130:44479): tablet 51d8cdfd9048409ca4643fd55b3b17c9 (table test-table [id=4de230ca6ab24386a6dfa9b51d5d571a]) successfully deleted
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:14.790150 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.777244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.3.6.129:46461)
0925 05:10:14.777286 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.777288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.779120 (+ 1832us) server_negotiation.cc:407] Connection header received
0925 05:10:14.779190 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.779191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.779203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.779270 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.780015 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.780772 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.781884 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.782187 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.782261 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.786288 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.786300 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.786300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.786310 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.788582 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.789043 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.789047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.789048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.789121 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.789363 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.789363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.789364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.789565 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.789718 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.789940 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.790044 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 05:10:14.790607 4410 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55624
I20260925 05:10:14.799646 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:14.791702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.3.6.190:36973)
0925 05:10:14.791771 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:14.791773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:14.791894 (+ 121us) server_negotiation.cc:407] Connection header received
0925 05:10:14.791984 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:14.791986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:14.791998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:14.792063 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:14.792698 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.793492 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.794499 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:14.794825 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:14.794905 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:14.796542 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:14.796544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:14.796544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:14.796552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:14.798300 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.798715 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.798716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.798716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.798784 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:14.799014 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:14.799015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:14.799015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:14.799165 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:14.799268 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:14.799443 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 05:10:14.799530 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 05:10:14.800011 4212 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36952
I20260925 05:10:14.801046 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 4285
I20260925 05:10:14.808988 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 4425
I20260925 05:10:14.814345 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 4565
I20260925 05:10:14.819928 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 4181
2026-09-25T05:10:14Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2505 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4782](info): set up 2 sockets
Sep 25 05:10:14 dist-test-slave-1lvv krb5kdc[4782](info): commencing operation
krb5kdc: starting...
W20260925 05:10:15.863847 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:15 dist-test-slave-1lvv krb5kdc[4782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:15.994971 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:37463
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:16.078445 4795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:16.078707 4795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:16.078745 4795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:16.079983 4795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:16.080047 4795 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:10:16.080068 4795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:16.080091 4795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:16.080121 4795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:16.080145 4795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:16.081619 4795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:37463
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:16.081897 4795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:16.082140 4795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:16.087544 4801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.087549 4800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.087622 4795 server_base.cc:1034] running on GCE node
W20260925 05:10:16.087558 4803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.088020 4795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:16.088295 4795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:16.089489 4795 hybrid_clock.cc:648] HybridClock initialized: now 1790313016089469 us; error 39 us; skew 500 ppm
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:16.092221 4795 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:16.092716 4795 webserver.cc:460] Webserver started at http://127.3.6.190:45557/ using document root <none> and password file <none>
I20260925 05:10:16.092964 4795 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:16.093014 4795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:16.093110 4795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:16.094162 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "f525118edfda4e7e85a3c0045cb6ded8"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "ec3aad24d9ec7745dfee602fe3d2795d"
server_key_iv: "1198c804bd3f01c963c043a4f35f6556"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.094480 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "f525118edfda4e7e85a3c0045cb6ded8"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "ec3aad24d9ec7745dfee602fe3d2795d"
server_key_iv: "1198c804bd3f01c963c043a4f35f6556"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.095944 4795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:16.096750 4810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.096969 4795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:16.097067 4795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "f525118edfda4e7e85a3c0045cb6ded8"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "ec3aad24d9ec7745dfee602fe3d2795d"
server_key_iv: "1198c804bd3f01c963c043a4f35f6556"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.097133 4795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:16.109114 4795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:16.111303 4795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:16.111531 4795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:16.116725 4795 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:37463
I20260925 05:10:16.116835 4872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:37463 every 8 connection(s)
I20260925 05:10:16.117200 4795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:16.117970 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.120411 4873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Bootstrap starting.
I20260925 05:10:16.120947 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 4795
I20260925 05:10:16.120980 4873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.121064 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:16.121250 3098 external_mini_cluster.cc:1428] Setting key c610870ef3c65d6ff5c44a05c9f85377
I20260925 05:10:16.121289 4873 log.cc:824] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:16.122197 4873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: No bootstrap required, opened a new log
I20260925 05:10:16.124293 4873 raft_consensus.cc:348] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525118edfda4e7e85a3c0045cb6ded8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37463 } }
I20260925 05:10:16.124465 4873 raft_consensus.cc:374] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.124527 4873 raft_consensus.cc:729] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f525118edfda4e7e85a3c0045cb6ded8, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.124673 4873 consensus_queue.cc:260] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525118edfda4e7e85a3c0045cb6ded8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37463 } }
I20260925 05:10:16.124753 4873 raft_consensus.cc:388] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:16.124797 4873 raft_consensus.cc:482] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:16.124845 4873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:16.125411 4873 raft_consensus.cc:504] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525118edfda4e7e85a3c0045cb6ded8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37463 } }
I20260925 05:10:16.125530 4873 leader_election.cc:302] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f525118edfda4e7e85a3c0045cb6ded8; no voters:
I20260925 05:10:16.125723 4873 leader_election.cc:288] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:16.125751 4878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:16.125869 4878 raft_consensus.cc:686] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [term 1 LEADER]: Becoming Leader. State: Replica: f525118edfda4e7e85a3c0045cb6ded8, State: Running, Role: LEADER
I20260925 05:10:16.125958 4878 consensus_queue.cc:237] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525118edfda4e7e85a3c0045cb6ded8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37463 } }
I20260925 05:10:16.126008 4873 sys_catalog.cc:564] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:16.126297 4878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f525118edfda4e7e85a3c0045cb6ded8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525118edfda4e7e85a3c0045cb6ded8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37463 } } }
I20260925 05:10:16.126377 4878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:16.126498 4878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f525118edfda4e7e85a3c0045cb6ded8. Latest consensus state: current_term: 1 leader_uuid: "f525118edfda4e7e85a3c0045cb6ded8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525118edfda4e7e85a3c0045cb6ded8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37463 } } }
I20260925 05:10:16.126559 4878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:16.126673 4882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:16.127497 4882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:16.129310 4882 catalog_manager.cc:1347] Generated new cluster ID: 8b81b92c18964da3b0189b490d0ecc31
I20260925 05:10:16.129369 4882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:16.132797 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.121814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.3.6.190:37463)
0925 05:10:16.122014 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.122024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.122060 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:10:16.122545 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.122552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.122606 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.122762 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.123393 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.124342 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.125291 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.125804 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.125929 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.129442 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.129447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.129448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.129461 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.131196 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.131590 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.131593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.131593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.131679 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.131934 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.131935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.131935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.132175 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.132351 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.132446 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.132588 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260925 05:10:16.135974 4882 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:16.136534 4882 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:16.144176 4882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Generated new TSK 0
I20260925 05:10:16.144418 4882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:16.179275 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37463
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:16.277993 4899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:16.278249 4899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:16.278295 4899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:16.279570 4899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:16.279639 4899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:16.279672 4899 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:10:16.279703 4899 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:16.279735 4899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:16.281335 4899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:37463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:16.281610 4899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:16.281903 4899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:16.287334 4907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.287328 4905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.287573 4899 server_base.cc:1034] running on GCE node
W20260925 05:10:16.287334 4904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.287922 4899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:16.288218 4899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:16.289386 4899 hybrid_clock.cc:648] HybridClock initialized: now 1790313016289370 us; error 46 us; skew 500 ppm
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:16.292250 4899 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:16.292696 4899 webserver.cc:460] Webserver started at http://127.3.6.129:37449/ using document root <none> and password file <none>
I20260925 05:10:16.292891 4899 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:16.292935 4899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:16.293079 4899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:16.294181 4899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "a56a2803a7d849a98d2c1c2eb8ee706b"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "e7c95880a339041eab8fdb7893477d15"
server_key_iv: "2efb827f6d46e4986a7e35a6fcd0bd5b"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.294516 4899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "a56a2803a7d849a98d2c1c2eb8ee706b"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "e7c95880a339041eab8fdb7893477d15"
server_key_iv: "2efb827f6d46e4986a7e35a6fcd0bd5b"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.295924 4899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:16.296741 4914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.296965 4899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:16.297056 4899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "a56a2803a7d849a98d2c1c2eb8ee706b"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "e7c95880a339041eab8fdb7893477d15"
server_key_iv: "2efb827f6d46e4986a7e35a6fcd0bd5b"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.297116 4899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:16.314607 4899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:16.317075 4899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:16.317200 4899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:16.317695 4899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:16.317737 4899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.317766 4899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:16.317821 4899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:16.325335 4899 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:33903
I20260925 05:10:16.325403 5029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:33903 every 8 connection(s)
I20260925 05:10:16.325835 4899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:16.326342 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 4899
I20260925 05:10:16.326462 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:16.326651 3098 external_mini_cluster.cc:1428] Setting key cde372aa89132e3481a5f152b96d573f
I20260925 05:10:16.329082 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.318354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:59033 (local address 127.3.6.190:37463)
0925 05:10:16.318428 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.318430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.318632 (+ 202us) server_negotiation.cc:407] Connection header received
0925 05:10:16.319292 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.319293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.319307 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.319389 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.320233 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.321114 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.322194 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.322415 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.322500 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.325255 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.325256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.325257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.325265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.327492 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.328024 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.328031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.328032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.328104 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.328347 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.328348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.328349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.328504 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.328570 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.328833 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.328937 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:10:16.329211 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.318323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37463 (local address 127.3.6.129:59033)
0925 05:10:16.318511 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.318554 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.319145 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.319473 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.319478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.319887 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.320147 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.320152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.321216 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.321218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.322098 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.322100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.322525 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.322527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.322662 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.323212 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.323228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.325130 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.327591 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.327599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.327602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.327930 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.328170 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.328171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.328171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.328276 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.328564 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.328569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.328698 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.328778 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.328907 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260925 05:10:16.331852 5030 heartbeater.cc:347] Connected to a master server at 127.3.6.190:37463
I20260925 05:10:16.331990 5030 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:16.332212 5030 heartbeater.cc:511] Master 127.3.6.190:37463 requested a full tablet report, sending...
I20260925 05:10:16.332831 4826 ts_manager.cc:194] Registered new tserver with Master: a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903)
I20260925 05:10:16.333431 4826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:59033
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
I20260925 05:10:16.338317 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.330788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:55059 (local address 127.3.6.190:37463)
0925 05:10:16.330830 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.330832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.330987 (+ 155us) server_negotiation.cc:407] Connection header received
0925 05:10:16.331071 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.331072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.331081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.331109 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.331446 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.332118 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.334212 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.334418 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.334492 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.335457 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.335459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.335459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.335466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.336937 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.337304 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.337305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.337306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.337367 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.337780 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.337781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.337781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.337892 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.337973 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.338135 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.338216 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:16.338331 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.330759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37463 (local address 127.3.6.129:55059)
0925 05:10:16.330917 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.330928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.331002 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.331166 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.331167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.331321 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.331373 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.331374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.332233 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.332234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.334125 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.334127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.334489 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.334490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.334563 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.335034 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.335040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.335351 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.337038 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.337040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.337040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.337219 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.337440 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.337440 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.337441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.337503 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.337978 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.337979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.338024 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.338107 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.338217 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:16.354907 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37463
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:16.442498 5039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:16.442752 5039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:16.442799 5039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:16.444022 5039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:16.444087 5039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:16.444123 5039 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:10:16.444151 5039 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:16.444185 5039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:16.445652 5039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:37463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:16.445919 5039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:16.446156 5039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:16.453665 5047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.453688 5044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.453926 5045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.455837 5039 server_base.cc:1034] running on GCE node
I20260925 05:10:16.456099 5039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:16.456387 5039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:16.457616 5039 hybrid_clock.cc:648] HybridClock initialized: now 1790313016457596 us; error 39 us; skew 500 ppm
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:16.460604 5039 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:16.461077 5039 webserver.cc:460] Webserver started at http://127.3.6.130:36967/ using document root <none> and password file <none>
I20260925 05:10:16.461349 5039 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:16.461421 5039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:16.461578 5039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:16.462692 5039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "939ffe76074d4448a821b9480fb3db7f"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "7e2e35e77a23ec4ddfabed65cc458248"
server_key_iv: "992ea2055bbd505474168742cecc27d6"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.463086 5039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "939ffe76074d4448a821b9480fb3db7f"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "7e2e35e77a23ec4ddfabed65cc458248"
server_key_iv: "992ea2055bbd505474168742cecc27d6"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.464672 5039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 05:10:16.465472 5054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.465642 5039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:16.465745 5039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "939ffe76074d4448a821b9480fb3db7f"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "7e2e35e77a23ec4ddfabed65cc458248"
server_key_iv: "992ea2055bbd505474168742cecc27d6"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.465830 5039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:16.481051 5039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:16.483431 5039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:16.483603 5039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:16.484257 5039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:16.484305 5039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.484332 5039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:16.484365 5039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:16.491542 5039 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:46781
I20260925 05:10:16.491571 5169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:46781 every 8 connection(s)
I20260925 05:10:16.492048 5039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:16.494745 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.484873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:36587 (local address 127.3.6.190:37463)
0925 05:10:16.484940 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.484942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.485167 (+ 225us) server_negotiation.cc:407] Connection header received
0925 05:10:16.485743 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.485745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.485760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.485842 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.486810 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.487696 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.488826 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.489198 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.489282 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.492059 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.492061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.492061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.492070 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.493608 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.493981 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.493982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.493982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.494037 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.494238 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.494239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.494239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.494343 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.494392 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.494619 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.494681 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:10:16.494978 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.484831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37463 (local address 127.3.6.130:36587)
0925 05:10:16.485032 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.485074 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.485623 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.485956 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.485961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.486373 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.486696 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.486704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.487828 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.487829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.488716 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.488718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.489307 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.489308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.489467 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.490128 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.490140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.491955 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.493683 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.493689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.493691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.493916 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.494091 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.494092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.494093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.494187 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.494393 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.494397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.494512 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.494597 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.494721 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260925 05:10:16.495473 5170 heartbeater.cc:347] Connected to a master server at 127.3.6.190:37463
I20260925 05:10:16.495559 5170 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:16.495790 5170 heartbeater.cc:511] Master 127.3.6.190:37463 requested a full tablet report, sending...
I20260925 05:10:16.496157 4827 ts_manager.cc:194] Registered new tserver with Master: 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781)
I20260925 05:10:16.496532 4827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:36587
I20260925 05:10:16.501185 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 5039
I20260925 05:10:16.501292 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:16.501489 3098 external_mini_cluster.cc:1428] Setting key 54041fcd5009c667f581c74fe66fa862
I20260925 05:10:16.502837 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.496659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:56111 (local address 127.3.6.190:37463)
0925 05:10:16.496696 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.496698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.497007 (+ 309us) server_negotiation.cc:407] Connection header received
0925 05:10:16.497065 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.497066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.497075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.497119 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.497455 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.497916 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.498575 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.498754 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.498813 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.499966 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.499968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.499968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.499974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.501647 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.502017 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.502019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.502019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.502081 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.502275 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.502276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.502276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.502431 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.502503 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.502662 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.502747 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:10:16.502853 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.496648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37463 (local address 127.3.6.130:56111)
0925 05:10:16.496936 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.496947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.497008 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.497124 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.497125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.497320 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.497380 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.497382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.497997 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.497998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.498484 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.498486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.498845 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.498846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.498916 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.499443 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.499449 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.499883 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.501722 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.501723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.501723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.501934 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.502146 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.502146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.502147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.502207 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.502505 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.502506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.502554 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.502633 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.502743 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:16.529106 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37463
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:16.618283 5179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:16.618594 5179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:16.618646 5179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:16.619966 5179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:16.620052 5179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:16.620093 5179 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:10:16.620147 5179 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:16.620186 5179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:16.621804 5179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:37463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:16.622102 5179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:16.622352 5179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:16.627638 5184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.627696 5185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:16.627743 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:16.627934 5179 server_base.cc:1034] running on GCE node
I20260925 05:10:16.628150 5179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:16.628402 5179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:16.629598 5179 hybrid_clock.cc:648] HybridClock initialized: now 1790313016629561 us; error 34 us; skew 500 ppm
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:16.632329 5179 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:16.632843 5179 webserver.cc:460] Webserver started at http://127.3.6.131:42267/ using document root <none> and password file <none>
I20260925 05:10:16.633068 5179 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:16.633136 5179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:16.633281 5179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:16.634346 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "5f567430bcb840f1988d3dd6e7c7d825"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "132352a520b48064922aba46f83630ce"
server_key_iv: "918b631f0073e02abbae5b9382c46820"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.634871 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "5f567430bcb840f1988d3dd6e7c7d825"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "132352a520b48064922aba46f83630ce"
server_key_iv: "918b631f0073e02abbae5b9382c46820"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.636505 5179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:10:16.637259 5194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.637426 5179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:16.637526 5179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "5f567430bcb840f1988d3dd6e7c7d825"
format_stamp: "Formatted at 2026-09-25 05:10:16 on dist-test-slave-1lvv"
server_key: "132352a520b48064922aba46f83630ce"
server_key_iv: "918b631f0073e02abbae5b9382c46820"
server_key_version: "encryptionkey@0"
I20260925 05:10:16.637601 5179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:16.648048 5179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:16.650048 5179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:16.650218 5179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:16.650792 5179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:16.650851 5179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.650897 5179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:16.650925 5179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:16.658712 5179 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:42655
I20260925 05:10:16.658844 5309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:42655 every 8 connection(s)
I20260925 05:10:16.659129 5179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:16.661979 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.651284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:44831 (local address 127.3.6.190:37463)
0925 05:10:16.651354 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.651357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.651659 (+ 302us) server_negotiation.cc:407] Connection header received
0925 05:10:16.652497 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.652498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.652518 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.652567 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.653593 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.654149 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.655941 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.656173 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.656278 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.658974 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.658976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.658976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.658987 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.660553 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.660976 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.660977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.660978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.661040 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.661325 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.661326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.661326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.661450 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.661524 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.661798 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.661886 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:10:16.662182 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.651250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37463 (local address 127.3.6.131:44831)
0925 05:10:16.651470 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.651532 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.652367 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.652659 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.652664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.653119 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.653472 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.653481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.654274 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.654275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.655849 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.655854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.656299 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.656301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.656430 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.657026 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.657035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.658849 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.660659 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.660666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.660668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.660894 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.661112 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.661112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.661113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.661231 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.661513 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.661520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.661667 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.661777 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.661942 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
I20260925 05:10:16.662647 5310 heartbeater.cc:347] Connected to a master server at 127.3.6.190:37463
I20260925 05:10:16.662742 5310 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:16.662918 5310 heartbeater.cc:511] Master 127.3.6.190:37463 requested a full tablet report, sending...
I20260925 05:10:16.663318 4826 ts_manager.cc:194] Registered new tserver with Master: 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655)
I20260925 05:10:16.663863 4826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:44831
I20260925 05:10:16.664942 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 5179
I20260925 05:10:16.665022 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:16.665234 3098 external_mini_cluster.cc:1428] Setting key 3909788f0a9eaa4eb800906cd21c1ae4
I20260925 05:10:16.666039 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:16.669934 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.663795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:56371 (local address 127.3.6.190:37463)
0925 05:10:16.663848 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.663850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.663931 (+ 81us) server_negotiation.cc:407] Connection header received
0925 05:10:16.664030 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.664031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.664042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.664094 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.664441 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.665027 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.665971 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.666242 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.666304 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.667161 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.667163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.667163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.667171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.668752 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.669210 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.669211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.669211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.669264 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.669494 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.669494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.669495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.669596 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.669647 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.669810 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.669871 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:10:16.670085 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.663706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37463 (local address 127.3.6.131:56371)
0925 05:10:16.663857 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.663868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.663946 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.664096 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.664097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.664301 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.664367 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.664369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.665140 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.665141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.665872 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.665875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.666313 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.666313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.666378 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.666809 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.666813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.667080 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.668834 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.668835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.668836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.669110 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.669348 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.669349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.669350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.669432 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.669660 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.669661 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.669710 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.669791 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.669909 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:16.690551 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.683418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.3.6.190:37463)
0925 05:10:16.683477 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.683480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.683490 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:16.683543 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.683544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.683587 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.683649 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.684331 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.684841 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.685605 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.685802 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.685854 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.688156 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.688157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.688157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.688165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.689368 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.689808 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.689809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.689810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.689870 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.690103 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.690103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.690104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.690217 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.690292 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.690386 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.690453 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:10:16.692322 4826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:16.694072 4826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:16.701054 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.697007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.3.6.131:42655)
0925 05:10:16.697317 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.697321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.697332 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:16.697374 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.697376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.697392 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.697465 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:16.697574 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.698293 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.699872 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.700549 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.700638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.700713 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.700896 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.700939 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
I20260925 05:10:16.701048 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.696784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42655 (local address 127.0.0.1:58726)
0925 05:10:16.697098 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.697106 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.697141 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.697463 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.697464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.697466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:16.697518 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.697520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.698385 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.698386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.699710 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.699713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.700636 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.700637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.700758 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.700780 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.700876 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.700942 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":85,"threads_started":1}
I20260925 05:10:16.701254 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.696620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.3.6.129:33903)
0925 05:10:16.696809 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.696816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.696953 (+ 137us) server_negotiation.cc:407] Connection header received
0925 05:10:16.697168 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.697171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.697189 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.697281 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:16.697550 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.698713 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.699895 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.700608 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.700682 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.700740 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.701099 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.701155 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260925 05:10:16.701249 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.696514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46781 (local address 127.0.0.1:57168)
0925 05:10:16.696924 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.696933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.696995 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.697360 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.697363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.697365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:16.697463 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.697466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.698779 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.698780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.699944 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.699946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.701026 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.701027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.701115 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.701118 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.701150 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.701185 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":6,"thread_start_us":81,"threads_started":1}
I20260925 05:10:16.701680 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.696615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.3.6.130:46781)
0925 05:10:16.696907 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.696911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.697189 (+ 278us) server_negotiation.cc:407] Connection header received
0925 05:10:16.697232 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.697235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.697250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.697545 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:16.697611 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.698681 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.700024 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.700937 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.701333 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.701480 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.701513 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.701572 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260925 05:10:16.701056 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.696514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:33903 (local address 127.0.0.1:37452)
0925 05:10:16.696818 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.696861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.696995 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.697359 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.697363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.697368 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:16.697463 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.697465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.698805 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.698806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.699794 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.699797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.700777 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.700778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.700845 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.700849 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.700903 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.700951 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260925 05:10:16.702256 5244 tablet_service.cc:1467] Processing CreateTablet for tablet c315faf3614046d7a224732a3e5521df (DEFAULT_TABLE table=test-table [id=8f02b0bfd2044897a7208c12e17f0024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:16.702631 5244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315faf3614046d7a224732a3e5521df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.702611 4964 tablet_service.cc:1467] Processing CreateTablet for tablet c315faf3614046d7a224732a3e5521df (DEFAULT_TABLE table=test-table [id=8f02b0bfd2044897a7208c12e17f0024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:16.702963 4964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315faf3614046d7a224732a3e5521df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.704264 5104 tablet_service.cc:1467] Processing CreateTablet for tablet c315faf3614046d7a224732a3e5521df (DEFAULT_TABLE table=test-table [id=8f02b0bfd2044897a7208c12e17f0024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:16.704620 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315faf3614046d7a224732a3e5521df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.705930 5333 tablet_bootstrap.cc:492] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Bootstrap starting.
I20260925 05:10:16.706048 5334 tablet_bootstrap.cc:492] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Bootstrap starting.
I20260925 05:10:16.706738 5334 tablet_bootstrap.cc:654] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.706877 5333 tablet_bootstrap.cc:654] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.707084 5334 log.cc:824] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:16.707119 5333 log.cc:824] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:16.707871 5333 tablet_bootstrap.cc:492] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: No bootstrap required, opened a new log
I20260925 05:10:16.707952 5333 ts_tablet_manager.cc:1397] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:16.708529 5337 tablet_bootstrap.cc:492] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Bootstrap starting.
I20260925 05:10:16.708726 5334 tablet_bootstrap.cc:492] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: No bootstrap required, opened a new log
I20260925 05:10:16.708803 5334 ts_tablet_manager.cc:1397] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 05:10:16.709198 5337 tablet_bootstrap.cc:654] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.709491 5337 log.cc:824] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:16.710197 5337 tablet_bootstrap.cc:492] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: No bootstrap required, opened a new log
I20260925 05:10:16.710276 5337 ts_tablet_manager.cc:1397] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:16.710155 5333 raft_consensus.cc:348] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.710325 5333 raft_consensus.cc:374] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.710369 5333 raft_consensus.cc:729] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.710503 5333 consensus_queue.cc:260] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.710804 5333 ts_tablet_manager.cc:1428] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:16.710832 5310 heartbeater.cc:503] Master 127.3.6.190:37463 was elected leader, sending a full tablet report...
I20260925 05:10:16.710883 5334 raft_consensus.cc:348] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.711014 5334 raft_consensus.cc:374] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.711035 5334 raft_consensus.cc:729] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56a2803a7d849a98d2c1c2eb8ee706b, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.711160 5334 consensus_queue.cc:260] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.711475 5334 ts_tablet_manager.cc:1428] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:16.711486 5030 heartbeater.cc:503] Master 127.3.6.190:37463 was elected leader, sending a full tablet report...
I20260925 05:10:16.712364 5337 raft_consensus.cc:348] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.712522 5337 raft_consensus.cc:374] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.712569 5337 raft_consensus.cc:729] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939ffe76074d4448a821b9480fb3db7f, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.712687 5337 consensus_queue.cc:260] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.712925 5337 ts_tablet_manager.cc:1428] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:16.712996 5170 heartbeater.cc:503] Master 127.3.6.190:37463 was elected leader, sending a full tablet report...
W20260925 05:10:16.742710 5171 tablet.cc:2369] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:16.752254 5341 raft_consensus.cc:482] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:16.752394 5341 raft_consensus.cc:504] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.752782 5341 leader_election.cc:288] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655)
I20260925 05:10:16.757153 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.753386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:35793 (local address 127.3.6.131:42655)
0925 05:10:16.753432 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.753434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.753583 (+ 149us) server_negotiation.cc:407] Connection header received
0925 05:10:16.753637 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.753638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.753647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.753717 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:16.753867 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.754567 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.756118 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.756687 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.756764 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.756777 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.757029 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.757074 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:16.757153 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.753219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:37927 (local address 127.3.6.129:33903)
0925 05:10:16.753303 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.753306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.753377 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:10:16.753461 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.753462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.753474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.753545 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:16.753772 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.754582 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.756117 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.756680 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.756764 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.756777 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.757029 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.757074 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:10:16.757181 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.753133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:33903 (local address 127.3.6.130:37927)
0925 05:10:16.753284 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.753298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.753385 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.753578 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.753579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.753581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:16.753663 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.753667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.754699 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.754700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.756010 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.756014 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.756828 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.756829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.756928 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.756940 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.757012 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.757070 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":41}
I20260925 05:10:16.757194 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.753289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42655 (local address 127.3.6.130:35793)
0925 05:10:16.753508 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.753517 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.753578 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.753724 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.753725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.753727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:16.753790 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.753792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.754699 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.754700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.756010 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.756014 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.756828 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.756829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.756928 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.756940 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.757012 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.757077 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":118,"thread_start_us":130,"threads_started":1}
I20260925 05:10:16.757674 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315faf3614046d7a224732a3e5521df" candidate_uuid: "939ffe76074d4448a821b9480fb3db7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" is_pre_election: true
I20260925 05:10:16.757874 4984 raft_consensus.cc:2454] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 939ffe76074d4448a821b9480fb3db7f in term 0.
I20260925 05:10:16.758154 5057 leader_election.cc:302] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939ffe76074d4448a821b9480fb3db7f, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:16.758133 5264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315faf3614046d7a224732a3e5521df" candidate_uuid: "939ffe76074d4448a821b9480fb3db7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f567430bcb840f1988d3dd6e7c7d825" is_pre_election: true
I20260925 05:10:16.758289 5341 raft_consensus.cc:2781] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:16.758297 5264 raft_consensus.cc:2454] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 939ffe76074d4448a821b9480fb3db7f in term 0.
I20260925 05:10:16.758355 5341 raft_consensus.cc:482] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:16.758379 5341 raft_consensus.cc:3037] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:16.759580 5341 raft_consensus.cc:504] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.759720 5341 leader_election.cc:288] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [CANDIDATE]: Term 1 election: Requested vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655)
I20260925 05:10:16.759903 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315faf3614046d7a224732a3e5521df" candidate_uuid: "939ffe76074d4448a821b9480fb3db7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b"
I20260925 05:10:16.759948 5264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315faf3614046d7a224732a3e5521df" candidate_uuid: "939ffe76074d4448a821b9480fb3db7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f567430bcb840f1988d3dd6e7c7d825"
I20260925 05:10:16.759996 4984 raft_consensus.cc:3037] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:16.760056 5264 raft_consensus.cc:3037] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:16.760746 4984 raft_consensus.cc:2454] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939ffe76074d4448a821b9480fb3db7f in term 1.
I20260925 05:10:16.760860 5264 raft_consensus.cc:2454] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939ffe76074d4448a821b9480fb3db7f in term 1.
I20260925 05:10:16.760955 5057 leader_election.cc:302] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939ffe76074d4448a821b9480fb3db7f, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:16.761059 5341 raft_consensus.cc:2781] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:16.761232 5341 raft_consensus.cc:686] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 1 LEADER]: Becoming Leader. State: Replica: 939ffe76074d4448a821b9480fb3db7f, State: Running, Role: LEADER
I20260925 05:10:16.761348 5341 consensus_queue.cc:237] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.762247 4826 catalog_manager.cc:5495] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 939ffe76074d4448a821b9480fb3db7f (127.3.6.130). New cstate: current_term: 1 leader_uuid: "939ffe76074d4448a821b9480fb3db7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
Sep 25 05:10:16 dist-test-slave-1lvv krb5kdc[4782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:16.816187 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.808600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37463 (local address 127.0.0.1:54432)
0925 05:10:16.808798 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.808812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.808913 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.809112 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.809112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.809468 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:16.809521 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.809524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.810239 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.810240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.810820 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.810822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.811208 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.811209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.811266 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.811890 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:16.811895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:16.813352 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:16.815066 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.815075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.815077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.815299 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.815493 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:16.815493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:16.815493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:16.815559 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:16.815859 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:16.815866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:16.815915 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.815974 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.816075 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260925 05:10:16.816231 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.808661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.3.6.190:37463)
0925 05:10:16.808760 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.808762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.808893 (+ 131us) server_negotiation.cc:407] Connection header received
0925 05:10:16.809009 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.809010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.809020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.809107 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:16.809659 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.810156 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.810924 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.811109 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.811207 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.813477 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:16.813479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:16.813479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:16.813486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:16.814963 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.815372 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.815372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.815373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.815429 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:16.815618 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:16.815619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:16.815619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:16.815734 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:16.815871 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.816012 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.816125 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":69}
I20260925 05:10:16.818629 4826 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:54432:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:10:16.818852 4826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:16.820520 4964 tablet_service.cc:1467] Processing CreateTablet for tablet d3419fabef8e4f639c75c797812d4740 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ff09ba282964dd686cf542353ce506a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:16.820549 5104 tablet_service.cc:1467] Processing CreateTablet for tablet d3419fabef8e4f639c75c797812d4740 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ff09ba282964dd686cf542353ce506a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:16.820715 4964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3419fabef8e4f639c75c797812d4740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.820734 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3419fabef8e4f639c75c797812d4740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.821914 5244 tablet_service.cc:1467] Processing CreateTablet for tablet d3419fabef8e4f639c75c797812d4740 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ff09ba282964dd686cf542353ce506a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:16.821978 5334 tablet_bootstrap.cc:492] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: Bootstrap starting.
I20260925 05:10:16.822041 5337 tablet_bootstrap.cc:492] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: Bootstrap starting.
I20260925 05:10:16.822095 5244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3419fabef8e4f639c75c797812d4740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.822463 5337 tablet_bootstrap.cc:654] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.822723 5334 tablet_bootstrap.cc:654] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.823374 5337 tablet_bootstrap.cc:492] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: No bootstrap required, opened a new log
I20260925 05:10:16.823446 5337 ts_tablet_manager.cc:1397] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:16.823603 5337 raft_consensus.cc:348] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.823720 5337 raft_consensus.cc:374] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.823715 5333 tablet_bootstrap.cc:492] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: Bootstrap starting.
I20260925 05:10:16.823747 5337 raft_consensus.cc:729] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939ffe76074d4448a821b9480fb3db7f, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.823829 5337 consensus_queue.cc:260] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.823939 5337 ts_tablet_manager.cc:1428] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:16.824306 5333 tablet_bootstrap.cc:654] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.824342 5334 tablet_bootstrap.cc:492] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: No bootstrap required, opened a new log
I20260925 05:10:16.824383 5334 ts_tablet_manager.cc:1397] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:16.824556 5334 raft_consensus.cc:348] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.824661 5334 raft_consensus.cc:374] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.824683 5334 raft_consensus.cc:729] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56a2803a7d849a98d2c1c2eb8ee706b, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.824738 5334 consensus_queue.cc:260] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.824841 5334 ts_tablet_manager.cc:1428] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:16.825009 5341 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.825119 5341 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:16.825184 5333 tablet_bootstrap.cc:492] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: No bootstrap required, opened a new log
I20260925 05:10:16.825238 5333 ts_tablet_manager.cc:1397] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:16.825274 5340 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.825389 5340 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:16.825397 5333 raft_consensus.cc:348] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.825479 5333 raft_consensus.cc:374] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.825501 5333 raft_consensus.cc:729] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.825562 5333 consensus_queue.cc:260] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.825657 5333 ts_tablet_manager.cc:1428] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.825788 5339 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.825856 5339 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 05:10:16.826802 5031 tablet.cc:2369] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:16.843956 5339 raft_consensus.cc:482] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:16.844089 5339 raft_consensus.cc:504] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.844486 5339 leader_election.cc:288] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781)
I20260925 05:10:16.847470 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.844689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:33903 (local address 127.3.6.131:52367)
0925 05:10:16.844745 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.844757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.844803 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.844961 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.844979 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.844982 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:16.845062 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.845064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.845776 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.845777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.846577 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.846580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.847155 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.847156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.847261 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.847268 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.847316 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.847382 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 05:10:16.847477 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.844757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:35165 (local address 127.3.6.130:46781)
0925 05:10:16.844812 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.844814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.844992 (+ 178us) server_negotiation.cc:407] Connection header received
0925 05:10:16.845043 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.845043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.845056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.845117 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:16.845275 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.845749 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.846728 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.847082 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.847175 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.847203 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.847376 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.847417 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:10:16.847470 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.844744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:52367 (local address 127.3.6.129:33903)
0925 05:10:16.844791 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.844793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.844803 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:16.844874 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.844875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.844886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.844942 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:16.845142 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.845691 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.846680 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.847071 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.847129 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.847160 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.847342 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.847388 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 05:10:16.847510 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.844722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46781 (local address 127.3.6.131:35165)
0925 05:10:16.844922 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.844931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.844981 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.845141 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.845142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.845143 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:16.845208 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.845209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.845838 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.845839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.846601 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.846603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.847224 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.847225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.847276 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.847279 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.847354 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.847393 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260925 05:10:16.848016 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3419fabef8e4f639c75c797812d4740" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f" is_pre_election: true
I20260925 05:10:16.848129 5124 raft_consensus.cc:2454] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 0.
I20260925 05:10:16.848342 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3419fabef8e4f639c75c797812d4740" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" is_pre_election: true
I20260925 05:10:16.848368 5198 leader_election.cc:302] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f567430bcb840f1988d3dd6e7c7d825, 939ffe76074d4448a821b9480fb3db7f; no voters:
I20260925 05:10:16.848449 4984 raft_consensus.cc:2454] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 0.
I20260925 05:10:16.848527 5339 raft_consensus.cc:2781] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:16.848583 5339 raft_consensus.cc:482] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:16.848614 5339 raft_consensus.cc:3037] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:16.849512 5339 raft_consensus.cc:504] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.849684 5339 leader_election.cc:288] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 election: Requested vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781)
I20260925 05:10:16.849874 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3419fabef8e4f639c75c797812d4740" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b"
I20260925 05:10:16.849957 4984 raft_consensus.cc:3037] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:16.849880 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3419fabef8e4f639c75c797812d4740" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f"
I20260925 05:10:16.849990 5124 raft_consensus.cc:3037] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:16.850596 4984 raft_consensus.cc:2454] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 1.
I20260925 05:10:16.850792 5197 leader_election.cc:302] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f567430bcb840f1988d3dd6e7c7d825, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:16.850821 5124 raft_consensus.cc:2454] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 1.
I20260925 05:10:16.850888 5339 raft_consensus.cc:2781] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:16.851066 5339 raft_consensus.cc:686] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 LEADER]: Becoming Leader. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Running, Role: LEADER
I20260925 05:10:16.851195 5339 consensus_queue.cc:237] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.851684 5355 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: TxnStatusTablet state changed. Reason: New leader 5f567430bcb840f1988d3dd6e7c7d825. Latest consensus state: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.851809 5355 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:16.852012 5357 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:10:16.852154 5357 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:10:16.852155 4827 catalog_manager.cc:5495] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20260925 05:10:16.862419 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.859529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.3.6.131:42655)
0925 05:10:16.859590 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.859592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.859604 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:16.859696 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.859697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.859713 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.859834 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:16.859977 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.860457 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.861596 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.861803 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.861856 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.862075 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:16.862169 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:16.862227 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.862338 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.862358 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:10:16.862479 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.859445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42655 (local address 127.0.0.1:58732)
0925 05:10:16.859537 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.859549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.859594 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.859845 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.859846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.859848 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:16.859907 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.859909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.860540 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.860541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.861468 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.861472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.861882 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.861883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.861956 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.861975 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:16.862264 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:16.862275 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.862336 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.862393 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:10:16.864451 4984 raft_consensus.cc:1261] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Refusing update from remote peer 5f567430bcb840f1988d3dd6e7c7d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:16.864451 5124 raft_consensus.cc:1261] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Refusing update from remote peer 5f567430bcb840f1988d3dd6e7c7d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:16.864810 5355 consensus_queue.cc:1035] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:16.864951 5339 consensus_queue.cc:1035] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:16.866235 5340 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: TxnStatusTablet state changed. Reason: New leader 5f567430bcb840f1988d3dd6e7c7d825. Latest consensus state: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.866338 5340 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:16.868451 5341 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: TxnStatusTablet state changed. Reason: New leader 5f567430bcb840f1988d3dd6e7c7d825. Latest consensus state: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.868491 5339 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.868575 5339 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:16.868574 5341 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:16.868623 5339 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.868675 5339 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P 5f567430bcb840f1988d3dd6e7c7d825: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:16.869271 5340 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.869380 5340 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P a56a2803a7d849a98d2c1c2eb8ee706b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:16.869603 5360 mvcc.cc:205] Tried to move back new op lower bound from 7333122117074092032 to 7333122117023182848. Current Snapshot: MvccSnapshot[applied={T|T < 7333122117074092032}]
I20260925 05:10:16.869606 5361 mvcc.cc:205] Tried to move back new op lower bound from 7333122117074092032 to 7333122117023182848. Current Snapshot: MvccSnapshot[applied={T|T < 7333122117074092032}]
I20260925 05:10:16.869401 5363 mvcc.cc:205] Tried to move back new op lower bound from 7333122117074092032 to 7333122117023182848. Current Snapshot: MvccSnapshot[applied={T|T < 7333122117074092032}]
I20260925 05:10:16.869911 5341 tablet_replica.cc:439] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } } }
I20260925 05:10:16.870002 5341 tablet_replica.cc:442] T d3419fabef8e4f639c75c797812d4740 P 939ffe76074d4448a821b9480fb3db7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:16.874169 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.871668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.3.6.130:46781)
0925 05:10:16.871734 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.871736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.871745 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:16.871801 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.871802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.871819 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.871906 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:16.872058 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.872546 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.873342 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.873557 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.873636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.873820 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:16.873932 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:16.874019 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.874062 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.874088 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:10:16.880007 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.877054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42655 (local address 127.3.6.130:54279)
0925 05:10:16.877133 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.877146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.877186 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.877337 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.877338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.877339 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:16.877396 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.877397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.878334 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.878335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.878960 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.878963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.879466 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.879467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.879525 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.879544 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:16.879836 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:16.879839 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.879903 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.879949 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 05:10:16.880016 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.877133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:54279 (local address 127.3.6.131:42655)
0925 05:10:16.877187 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.877189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.877231 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:10:16.877258 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.877259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.877272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.877328 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:16.877458 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.878211 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.879064 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.879377 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.879455 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.879674 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:16.879716 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:16.879816 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.879898 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.879923 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:10:16.882747 4984 raft_consensus.cc:1261] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Refusing update from remote peer 939ffe76074d4448a821b9480fb3db7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:16.882861 5264 raft_consensus.cc:1261] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Refusing update from remote peer 939ffe76074d4448a821b9480fb3db7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:16.883090 5341 consensus_queue.cc:1035] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:16.883284 5341 consensus_queue.cc:1035] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:16.893788 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.890109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46781 (local address 127.3.6.131:45397)
0925 05:10:16.890164 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:16.890178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:16.890229 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:16.890346 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:16.890347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:16.890348 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:16.890424 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.890426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.891033 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.891034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.892047 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:16.892051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.892724 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.892725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:16.892799 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.892817 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:16.893597 (+ 780us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:16.893599 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:16.893644 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 05:10:16.893697 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260925 05:10:16.894007 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:16.890201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:45397 (local address 127.3.6.130:46781)
0925 05:10:16.890239 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:16.890241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:16.890253 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:16.890277 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:16.890278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:16.890296 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:16.890341 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:16.890487 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.890950 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.892160 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:16.892598 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:16.892683 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:16.893439 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:16.893482 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:16.893544 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:16.893911 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 05:10:16.893937 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:10:16.902619 4827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:16.902765 4827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:16.903643 4827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Sending DeleteTablet for 3 replicas of tablet c315faf3614046d7a224732a3e5521df
I20260925 05:10:16.903996 5104 tablet_service.cc:1514] Processing DeleteTablet for tablet c315faf3614046d7a224732a3e5521df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:16 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:57168
I20260925 05:10:16.904047 5244 tablet_service.cc:1514] Processing DeleteTablet for tablet c315faf3614046d7a224732a3e5521df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:16 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58726
I20260925 05:10:16.904270 4964 tablet_service.cc:1514] Processing DeleteTablet for tablet c315faf3614046d7a224732a3e5521df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:16 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:37452
I20260925 05:10:16.904528 5380 tablet_replica.cc:330] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: stopping tablet replica
I20260925 05:10:16.904706 5380 raft_consensus.cc:2229] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:16.904840 5380 raft_consensus.cc:2258] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:16.904788 4827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:16.904940 4827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:16.905721 5380 ts_tablet_manager.cc:1905] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:16.906567 5244 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5c614b39f44691a2d1e28d8616f6c7 (DEFAULT_TABLE table=test-table [id=0a24e7d57136496e905ea13060f5db1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:16.906715 5244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5c614b39f44691a2d1e28d8616f6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.908171 5380 ts_tablet_manager.cc:1918] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:16.908337 5380 log.cc:1196] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/c315faf3614046d7a224732a3e5521df
I20260925 05:10:16.908450 5333 tablet_bootstrap.cc:492] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: Bootstrap starting.
I20260925 05:10:16.908636 4964 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5c614b39f44691a2d1e28d8616f6c7 (DEFAULT_TABLE table=test-table [id=0a24e7d57136496e905ea13060f5db1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:16.908746 4964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5c614b39f44691a2d1e28d8616f6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.908880 5380 ts_tablet_manager.cc:1939] T c315faf3614046d7a224732a3e5521df P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting consensus metadata
I20260925 05:10:16.908965 5333 tablet_bootstrap.cc:654] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.909394 4813 catalog_manager.cc:4841] TS a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903): tablet c315faf3614046d7a224732a3e5521df (table test-table [id=8f02b0bfd2044897a7208c12e17f0024]) successfully deleted
I20260925 05:10:16.909967 5334 tablet_bootstrap.cc:492] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: Bootstrap starting.
I20260925 05:10:16.910213 5104 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5c614b39f44691a2d1e28d8616f6c7 (DEFAULT_TABLE table=test-table [id=0a24e7d57136496e905ea13060f5db1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:16.910322 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5c614b39f44691a2d1e28d8616f6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:16.910374 5334 tablet_bootstrap.cc:654] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.911218 5334 tablet_bootstrap.cc:492] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: No bootstrap required, opened a new log
I20260925 05:10:16.911266 5334 ts_tablet_manager.cc:1397] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:16.911412 5334 raft_consensus.cc:348] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.911477 5334 raft_consensus.cc:374] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.911497 5334 raft_consensus.cc:729] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56a2803a7d849a98d2c1c2eb8ee706b, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.911537 5334 consensus_queue.cc:260] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.911616 5334 ts_tablet_manager.cc:1428] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:16.911914 5337 tablet_bootstrap.cc:492] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: Bootstrap starting.
I20260925 05:10:16.912390 5337 tablet_bootstrap.cc:654] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:16.913141 5337 tablet_bootstrap.cc:492] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: No bootstrap required, opened a new log
I20260925 05:10:16.913198 5337 ts_tablet_manager.cc:1397] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:16.913362 5337 raft_consensus.cc:348] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.913455 5337 raft_consensus.cc:374] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.913491 5337 raft_consensus.cc:729] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939ffe76074d4448a821b9480fb3db7f, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.913537 5337 consensus_queue.cc:260] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.913631 5337 ts_tablet_manager.cc:1428] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:16.913933 5378 tablet_replica.cc:330] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: stopping tablet replica
I20260925 05:10:16.914053 5378 raft_consensus.cc:2229] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:16.914206 5378 raft_consensus.cc:2258] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:16.914625 5378 ts_tablet_manager.cc:1905] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:16.916383 5378 ts_tablet_manager.cc:1918] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:16.916531 5378 log.cc:1196] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/c315faf3614046d7a224732a3e5521df
I20260925 05:10:16.916816 5378 ts_tablet_manager.cc:1939] T c315faf3614046d7a224732a3e5521df P 939ffe76074d4448a821b9480fb3db7f: Deleting consensus metadata
I20260925 05:10:16.917250 4814 catalog_manager.cc:4841] TS 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781): tablet c315faf3614046d7a224732a3e5521df (table test-table [id=8f02b0bfd2044897a7208c12e17f0024]) successfully deleted
I20260925 05:10:16.919891 5379 tablet_replica.cc:330] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: stopping tablet replica
I20260925 05:10:16.920044 5379 raft_consensus.cc:2229] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:16.920121 5379 raft_consensus.cc:2258] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:16.920275 5333 tablet_bootstrap.cc:492] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: No bootstrap required, opened a new log
I20260925 05:10:16.920331 5333 ts_tablet_manager.cc:1397] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260925 05:10:16.920702 5333 raft_consensus.cc:348] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.920764 5379 ts_tablet_manager.cc:1905] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:16.920815 5333 raft_consensus.cc:374] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:16.920840 5333 raft_consensus.cc:729] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Initialized, Role: FOLLOWER
I20260925 05:10:16.920882 5333 consensus_queue.cc:260] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:16.920979 5333 ts_tablet_manager.cc:1428] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:10:16.922719 5379 ts_tablet_manager.cc:1918] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:16.922797 5379 log.cc:1196] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/c315faf3614046d7a224732a3e5521df
I20260925 05:10:16.923051 5379 ts_tablet_manager.cc:1939] T c315faf3614046d7a224732a3e5521df P 5f567430bcb840f1988d3dd6e7c7d825: Deleting consensus metadata
I20260925 05:10:16.923470 4813 catalog_manager.cc:4841] TS 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655): tablet c315faf3614046d7a224732a3e5521df (table test-table [id=8f02b0bfd2044897a7208c12e17f0024]) successfully deleted
I20260925 05:10:17.129210 5365 raft_consensus.cc:482] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:17.129318 5365 raft_consensus.cc:504] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:17.129495 5365 leader_election.cc:288] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781)
I20260925 05:10:17.129796 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5c614b39f44691a2d1e28d8616f6c7" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" is_pre_election: true
I20260925 05:10:17.129798 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5c614b39f44691a2d1e28d8616f6c7" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f" is_pre_election: true
I20260925 05:10:17.129904 4984 raft_consensus.cc:2454] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 0.
I20260925 05:10:17.129904 5124 raft_consensus.cc:2454] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 0.
I20260925 05:10:17.130112 5197 leader_election.cc:302] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f567430bcb840f1988d3dd6e7c7d825, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:17.130261 5365 raft_consensus.cc:2781] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:17.130312 5365 raft_consensus.cc:482] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:17.130328 5365 raft_consensus.cc:3037] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:17.131026 5365 raft_consensus.cc:504] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:17.131155 5365 leader_election.cc:288] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 election: Requested vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781)
I20260925 05:10:17.131315 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5c614b39f44691a2d1e28d8616f6c7" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b"
I20260925 05:10:17.131333 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5c614b39f44691a2d1e28d8616f6c7" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f"
I20260925 05:10:17.131415 5124 raft_consensus.cc:3037] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:17.131419 4984 raft_consensus.cc:3037] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:17.132121 4984 raft_consensus.cc:2454] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 1.
I20260925 05:10:17.132140 5124 raft_consensus.cc:2454] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 1.
I20260925 05:10:17.132337 5197 leader_election.cc:302] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f567430bcb840f1988d3dd6e7c7d825, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:17.132450 5365 raft_consensus.cc:2781] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:17.132519 5365 raft_consensus.cc:686] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 LEADER]: Becoming Leader. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Running, Role: LEADER
I20260925 05:10:17.132596 5365 consensus_queue.cc:237] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:17.133206 4827 catalog_manager.cc:5495] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20260925 05:10:17.146245 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.144345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.3.6.131:42655)
0925 05:10:17.144389 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.144391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.144401 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:17.144457 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.144458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.144466 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.144504 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:17.144603 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.145067 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.145752 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.145931 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.145985 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.146087 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:17.146111 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:17.146144 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.146174 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.146186 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:17.150528 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.148595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:45693 (local address 127.3.6.131:42655)
0925 05:10:17.148649 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:17.148651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:17.148658 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:10:17.148677 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:17.148678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:17.148687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:17.148740 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:17.148830 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.149264 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.149951 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.150125 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.150181 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.150323 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:17.150345 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:17.150383 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:17.150442 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 05:10:17.150454 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:10:17.150532 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:17.148405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42655 (local address 127.3.6.131:45693)
0925 05:10:17.148562 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:17.148573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:17.148624 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:17.148746 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:17.148747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:17.148748 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:17.148781 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.148782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.149336 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.149337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.149877 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:17.149879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:17.150209 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:17.150210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:17.150253 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:17.150257 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:17.150397 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:17.150399 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:17.150434 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 05:10:17.150469 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 05:10:17.152807 4984 raft_consensus.cc:1261] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Refusing update from remote peer 5f567430bcb840f1988d3dd6e7c7d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:17.152769 5124 raft_consensus.cc:1261] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Refusing update from remote peer 5f567430bcb840f1988d3dd6e7c7d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:17.153038 5339 consensus_queue.cc:1035] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:17.153153 5355 consensus_queue.cc:1035] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:10:17.157734 5360 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:10:18.681710 4825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:18.681840 4825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:18.682802 4825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Sending DeleteTablet for 3 replicas of tablet 1f5c614b39f44691a2d1e28d8616f6c7
I20260925 05:10:18.683087 4964 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5c614b39f44691a2d1e28d8616f6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:18 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:37452
I20260925 05:10:18.683099 5244 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5c614b39f44691a2d1e28d8616f6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:18 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58726
I20260925 05:10:18.683238 5104 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5c614b39f44691a2d1e28d8616f6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:18 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:57168
I20260925 05:10:18.683338 5410 tablet_replica.cc:330] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: stopping tablet replica
I20260925 05:10:18.683403 5410 raft_consensus.cc:2229] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:18.683471 5411 tablet_replica.cc:330] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: stopping tablet replica
I20260925 05:10:18.683508 5410 raft_consensus.cc:2258] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:18.683535 5411 raft_consensus.cc:2229] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:18.683579 5411 raft_consensus.cc:2258] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:18.683815 5410 ts_tablet_manager.cc:1905] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:18.685735 5410 ts_tablet_manager.cc:1918] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:10:18.685803 5410 log.cc:1196] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/1f5c614b39f44691a2d1e28d8616f6c7
I20260925 05:10:18.686045 5410 ts_tablet_manager.cc:1939] T 1f5c614b39f44691a2d1e28d8616f6c7 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting consensus metadata
I20260925 05:10:18.686357 5409 tablet_replica.cc:330] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: stopping tablet replica
I20260925 05:10:18.686429 5409 raft_consensus.cc:2229] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:18.686479 5409 raft_consensus.cc:2258] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:18.686666 5411 ts_tablet_manager.cc:1905] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:18.686741 4813 catalog_manager.cc:4841] TS 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655): tablet 1f5c614b39f44691a2d1e28d8616f6c7 (table test-table [id=0a24e7d57136496e905ea13060f5db1c]) successfully deleted
I20260925 05:10:18.686764 5409 ts_tablet_manager.cc:1905] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:18.688124 5411 ts_tablet_manager.cc:1918] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:10:18.688136 5409 ts_tablet_manager.cc:1918] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:10:18.688205 5411 log.cc:1196] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/1f5c614b39f44691a2d1e28d8616f6c7
I20260925 05:10:18.688205 5409 log.cc:1196] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/1f5c614b39f44691a2d1e28d8616f6c7
I20260925 05:10:18.688436 5409 ts_tablet_manager.cc:1939] T 1f5c614b39f44691a2d1e28d8616f6c7 P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting consensus metadata
I20260925 05:10:18.688436 5411 ts_tablet_manager.cc:1939] T 1f5c614b39f44691a2d1e28d8616f6c7 P 939ffe76074d4448a821b9480fb3db7f: Deleting consensus metadata
I20260925 05:10:18.688778 4814 catalog_manager.cc:4841] TS 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781): tablet 1f5c614b39f44691a2d1e28d8616f6c7 (table test-table [id=0a24e7d57136496e905ea13060f5db1c]) successfully deleted
I20260925 05:10:18.688787 4813 catalog_manager.cc:4841] TS a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903): tablet 1f5c614b39f44691a2d1e28d8616f6c7 (table test-table [id=0a24e7d57136496e905ea13060f5db1c]) successfully deleted
Sep 25 05:10:18 dist-test-slave-1lvv krb5kdc[4782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313018, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:10:18 dist-test-slave-1lvv krb5kdc[4782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313018, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:18.713972 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.703261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.3.6.190:37463)
0925 05:10:18.703414 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.703416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.703428 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:18.703480 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.703481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.703490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.703554 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:18.704226 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.704737 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.705450 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.705643 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.705695 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.711413 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:18.711416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:18.711416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:18.711423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:18.712835 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:18.713225 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:18.713225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:18.713226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:18.713282 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:18.713458 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:18.713459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:18.713459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:18.713569 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:18.713647 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:18.713826 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 05:10:18.713896 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260925 05:10:18.715544 4825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:18.715703 4825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:18.717116 4964 tablet_service.cc:1467] Processing CreateTablet for tablet fc468c673df941608b03265a57e5a072 (DEFAULT_TABLE table=test-table [id=05f4804f5b0349399c646cf257e20e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:18.717126 5104 tablet_service.cc:1467] Processing CreateTablet for tablet fc468c673df941608b03265a57e5a072 (DEFAULT_TABLE table=test-table [id=05f4804f5b0349399c646cf257e20e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:18.717157 5244 tablet_service.cc:1467] Processing CreateTablet for tablet fc468c673df941608b03265a57e5a072 (DEFAULT_TABLE table=test-table [id=05f4804f5b0349399c646cf257e20e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:18.717278 4964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc468c673df941608b03265a57e5a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:18.717278 5244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc468c673df941608b03265a57e5a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:18.717295 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc468c673df941608b03265a57e5a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:18.718602 5420 tablet_bootstrap.cc:492] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: Bootstrap starting.
I20260925 05:10:18.718815 5422 tablet_bootstrap.cc:492] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: Bootstrap starting.
I20260925 05:10:18.719125 5420 tablet_bootstrap.cc:654] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:18.719347 5422 tablet_bootstrap.cc:654] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:18.719673 5421 tablet_bootstrap.cc:492] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: Bootstrap starting.
I20260925 05:10:18.719863 5420 tablet_bootstrap.cc:492] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: No bootstrap required, opened a new log
I20260925 05:10:18.719921 5420 ts_tablet_manager.cc:1397] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:18.720067 5421 tablet_bootstrap.cc:654] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:18.720045 5420 raft_consensus.cc:348] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.720149 5420 raft_consensus.cc:374] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:18.720172 5420 raft_consensus.cc:729] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939ffe76074d4448a821b9480fb3db7f, State: Initialized, Role: FOLLOWER
I20260925 05:10:18.720232 5420 consensus_queue.cc:260] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.720428 5420 ts_tablet_manager.cc:1428] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:18.720479 5422 tablet_bootstrap.cc:492] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: No bootstrap required, opened a new log
I20260925 05:10:18.720525 5422 ts_tablet_manager.cc:1397] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:18.720630 5422 raft_consensus.cc:348] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.720709 5422 raft_consensus.cc:374] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:18.720724 5422 raft_consensus.cc:729] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56a2803a7d849a98d2c1c2eb8ee706b, State: Initialized, Role: FOLLOWER
I20260925 05:10:18.720804 5422 consensus_queue.cc:260] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.720978 5422 ts_tablet_manager.cc:1428] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.721231 5421 tablet_bootstrap.cc:492] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: No bootstrap required, opened a new log
I20260925 05:10:18.721292 5421 ts_tablet_manager.cc:1397] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:18.721400 5421 raft_consensus.cc:348] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.721472 5421 raft_consensus.cc:374] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:18.721503 5421 raft_consensus.cc:729] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Initialized, Role: FOLLOWER
I20260925 05:10:18.721553 5421 consensus_queue.cc:260] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.721652 5421 ts_tablet_manager.cc:1428] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.823968 5427 raft_consensus.cc:482] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:18.824074 5427 raft_consensus.cc:504] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.824375 5427 leader_election.cc:288] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781), 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655)
I20260925 05:10:18.828040 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.824698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:39203 (local address 127.3.6.131:42655)
0925 05:10:18.824881 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.824883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.825117 (+ 234us) server_negotiation.cc:407] Connection header received
0925 05:10:18.825149 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.825150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.825162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.825237 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:18.825431 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.825924 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.827106 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.827705 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.827801 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.827814 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:18.827951 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 05:10:18.827980 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260925 05:10:18.828085 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.824624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42655 (local address 127.3.6.129:39203)
0925 05:10:18.824963 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:18.824976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:18.825065 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:18.825268 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:18.825269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:18.825272 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:18.825356 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.825357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.826021 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.826023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.826986 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.826989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.827799 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.827800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.827874 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.827879 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:18.827941 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 05:10:18.827979 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260925 05:10:18.828357 5264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468c673df941608b03265a57e5a072" candidate_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f567430bcb840f1988d3dd6e7c7d825" is_pre_election: true
I20260925 05:10:18.828446 5264 raft_consensus.cc:2454] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a56a2803a7d849a98d2c1c2eb8ee706b in term 0.
I20260925 05:10:18.828572 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.824624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46781 (local address 127.3.6.129:49363)
0925 05:10:18.824812 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:18.824822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:18.824874 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:18.825102 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:18.825104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:18.825106 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:18.825202 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.825205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.825918 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.825919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.827481 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.827484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.828300 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.828301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.828371 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.828374 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:18.828454 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:10:18.828493 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260925 05:10:18.828572 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.824795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:49363 (local address 127.3.6.130:46781)
0925 05:10:18.824949 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.824951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.824960 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:18.824994 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.824996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.825009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.825086 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:18.825287 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.825835 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.827573 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.828196 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.828297 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.828308 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:18.828462 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:10:18.828502 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260925 05:10:18.828622 4917 leader_election.cc:302] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f567430bcb840f1988d3dd6e7c7d825, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:18.828778 5427 raft_consensus.cc:2781] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:18.828820 5427 raft_consensus.cc:482] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:18.828842 5427 raft_consensus.cc:3037] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:18.828833 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468c673df941608b03265a57e5a072" candidate_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f" is_pre_election: true
I20260925 05:10:18.828920 5124 raft_consensus.cc:2454] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a56a2803a7d849a98d2c1c2eb8ee706b in term 0.
I20260925 05:10:18.829649 5427 raft_consensus.cc:504] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.829802 5427 leader_election.cc:288] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [CANDIDATE]: Term 1 election: Requested vote from peers 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781), 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655)
I20260925 05:10:18.829941 5264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468c673df941608b03265a57e5a072" candidate_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f567430bcb840f1988d3dd6e7c7d825"
I20260925 05:10:18.829967 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc468c673df941608b03265a57e5a072" candidate_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f"
I20260925 05:10:18.830040 5264 raft_consensus.cc:3037] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:18.830045 5124 raft_consensus.cc:3037] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:18.830722 5124 raft_consensus.cc:2454] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a56a2803a7d849a98d2c1c2eb8ee706b in term 1.
I20260925 05:10:18.830827 5264 raft_consensus.cc:2454] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a56a2803a7d849a98d2c1c2eb8ee706b in term 1.
I20260925 05:10:18.830914 4918 leader_election.cc:302] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939ffe76074d4448a821b9480fb3db7f, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:18.831053 5427 raft_consensus.cc:2781] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:18.831192 5427 raft_consensus.cc:686] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 LEADER]: Becoming Leader. State: Replica: a56a2803a7d849a98d2c1c2eb8ee706b, State: Running, Role: LEADER
I20260925 05:10:18.831272 5427 consensus_queue.cc:237] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.832023 4827 catalog_manager.cc:5495] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b reported cstate change: term changed from 0 to 1, leader changed from <none> to a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129). New cstate: current_term: 1 leader_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:18.841308 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.838840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.3.6.129:33903)
0925 05:10:18.838989 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.838991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.838999 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:10:18.839041 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.839041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.839061 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.839128 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:18.839238 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.839719 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.840596 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.840794 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.840847 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.840983 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:18.841090 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:18.841204 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:18.841219 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 05:10:18.841237 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260925 05:10:18.845809 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.843837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:42549 (local address 127.3.6.131:42655)
0925 05:10:18.843890 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.843892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.843929 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:10:18.843948 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.843949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.843960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.844001 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:18.844122 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.844557 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.845252 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.845427 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.845475 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.845612 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:18.845635 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:18.845674 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:18.845739 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 05:10:18.845756 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:10:18.845829 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.843630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42655 (local address 127.3.6.129:42549)
0925 05:10:18.843834 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:18.843844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:18.843889 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:18.844011 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:18.844011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:18.844013 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:18.844066 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.844067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.844613 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.844614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.845174 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.845175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.845486 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.845487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.845535 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.845549 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:18.845693 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:18.845694 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:10:18.845728 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 05:10:18.845759 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260925 05:10:18.848067 5264 raft_consensus.cc:1261] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Refusing update from remote peer a56a2803a7d849a98d2c1c2eb8ee706b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:18.848068 5124 raft_consensus.cc:1261] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Refusing update from remote peer a56a2803a7d849a98d2c1c2eb8ee706b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:18.848361 5432 consensus_queue.cc:1035] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:18.848493 5427 consensus_queue.cc:1035] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [LEADER]: Connected to new peer: Peer: permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:18.856088 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.853746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:32845 (local address 127.3.6.129:33903)
0925 05:10:18.853792 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.853794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.853982 (+ 188us) server_negotiation.cc:407] Connection header received
0925 05:10:18.854006 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.854006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.854016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.854056 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:18.854148 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.854607 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.855323 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.855582 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.855633 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.855844 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:18.855880 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:18.855933 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:18.856002 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:10:18.856024 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:10:18.856103 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.853705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:33903 (local address 127.3.6.131:32845)
0925 05:10:18.853856 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:18.853882 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:18.853947 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:18.854067 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:18.854067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:18.854069 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:18.854112 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.854114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.854673 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.854674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.855241 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:18.855242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.855639 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.855640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:18.855709 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.855714 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:18.855935 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:18.855937 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:18.855982 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 05:10:18.856035 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260925 05:10:18.862962 4825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:18.863085 4825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:18.863737 4825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Sending DeleteTablet for 3 replicas of tablet fc468c673df941608b03265a57e5a072
I20260925 05:10:18.864022 5104 tablet_service.cc:1514] Processing DeleteTablet for tablet fc468c673df941608b03265a57e5a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:18 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:57168
I20260925 05:10:18.864100 4964 tablet_service.cc:1514] Processing DeleteTablet for tablet fc468c673df941608b03265a57e5a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:18 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:37452
I20260925 05:10:18.864156 5411 tablet_replica.cc:330] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: stopping tablet replica
I20260925 05:10:18.864216 5411 raft_consensus.cc:2229] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:18.864224 5409 tablet_replica.cc:330] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: stopping tablet replica
I20260925 05:10:18.864264 5411 raft_consensus.cc:2258] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:18.864254 5244 tablet_service.cc:1514] Processing DeleteTablet for tablet fc468c673df941608b03265a57e5a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:18 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58726
I20260925 05:10:18.864279 5409 raft_consensus.cc:2229] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:18.864344 5410 tablet_replica.cc:330] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: stopping tablet replica
I20260925 05:10:18.864364 5409 raft_consensus.cc:2258] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:18.864406 5410 raft_consensus.cc:2229] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:18.864441 5410 raft_consensus.cc:2258] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:18.864612 4825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:18.864779 4825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:18.864823 5409 ts_tablet_manager.cc:1905] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:18.864853 5410 ts_tablet_manager.cc:1905] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:18.864899 5411 ts_tablet_manager.cc:1905] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:18.866480 5410 ts_tablet_manager.cc:1918] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:18.866544 5410 log.cc:1196] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/fc468c673df941608b03265a57e5a072
I20260925 05:10:18.866637 5411 ts_tablet_manager.cc:1918] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:18.866677 5409 ts_tablet_manager.cc:1918] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:18.866729 5409 log.cc:1196] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/fc468c673df941608b03265a57e5a072
I20260925 05:10:18.866729 5411 log.cc:1196] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/fc468c673df941608b03265a57e5a072
I20260925 05:10:18.866789 5410 ts_tablet_manager.cc:1939] T fc468c673df941608b03265a57e5a072 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting consensus metadata
I20260925 05:10:18.866976 5411 ts_tablet_manager.cc:1939] T fc468c673df941608b03265a57e5a072 P 939ffe76074d4448a821b9480fb3db7f: Deleting consensus metadata
I20260925 05:10:18.866986 5409 ts_tablet_manager.cc:1939] T fc468c673df941608b03265a57e5a072 P a56a2803a7d849a98d2c1c2eb8ee706b: Deleting consensus metadata
I20260925 05:10:18.867302 4813 catalog_manager.cc:4841] TS 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655): tablet fc468c673df941608b03265a57e5a072 (table test-table [id=05f4804f5b0349399c646cf257e20e5d]) successfully deleted
I20260925 05:10:18.867369 4813 catalog_manager.cc:4841] TS a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903): tablet fc468c673df941608b03265a57e5a072 (table test-table [id=05f4804f5b0349399c646cf257e20e5d]) successfully deleted
I20260925 05:10:18.867555 4814 catalog_manager.cc:4841] TS 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781): tablet fc468c673df941608b03265a57e5a072 (table test-table [id=05f4804f5b0349399c646cf257e20e5d]) successfully deleted
I20260925 05:10:18.868191 4964 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb3c280b975437ba6de7a31857a8c67 (DEFAULT_TABLE table=test-table [id=6db69b8b17994db6bfb5553a8813c93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:18.868201 5104 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb3c280b975437ba6de7a31857a8c67 (DEFAULT_TABLE table=test-table [id=6db69b8b17994db6bfb5553a8813c93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:18.868319 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb3c280b975437ba6de7a31857a8c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:18.868319 4964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb3c280b975437ba6de7a31857a8c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:18.869334 5420 tablet_bootstrap.cc:492] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: Bootstrap starting.
I20260925 05:10:18.869611 5422 tablet_bootstrap.cc:492] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b: Bootstrap starting.
I20260925 05:10:18.869670 5244 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb3c280b975437ba6de7a31857a8c67 (DEFAULT_TABLE table=test-table [id=6db69b8b17994db6bfb5553a8813c93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:18.869707 5420 tablet_bootstrap.cc:654] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:18.869874 5244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb3c280b975437ba6de7a31857a8c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:18.870072 5422 tablet_bootstrap.cc:654] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:18.870787 5422 tablet_bootstrap.cc:492] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b: No bootstrap required, opened a new log
I20260925 05:10:18.870841 5422 ts_tablet_manager.cc:1397] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:18.870896 5420 tablet_bootstrap.cc:492] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: No bootstrap required, opened a new log
I20260925 05:10:18.870940 5420 ts_tablet_manager.cc:1397] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:10:18.871013 5422 raft_consensus.cc:348] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.871081 5422 raft_consensus.cc:374] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:18.871047 5420 raft_consensus.cc:348] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.871104 5422 raft_consensus.cc:729] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56a2803a7d849a98d2c1c2eb8ee706b, State: Initialized, Role: FOLLOWER
I20260925 05:10:18.871106 5420 raft_consensus.cc:374] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:18.871130 5420 raft_consensus.cc:729] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939ffe76074d4448a821b9480fb3db7f, State: Initialized, Role: FOLLOWER
I20260925 05:10:18.871151 5422 consensus_queue.cc:260] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.871173 5420 consensus_queue.cc:260] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.871218 5421 tablet_bootstrap.cc:492] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Bootstrap starting.
I20260925 05:10:18.871258 5422 ts_tablet_manager.cc:1428] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.871279 5420 ts_tablet_manager.cc:1428] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:18.871536 5421 tablet_bootstrap.cc:654] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:18.872612 5421 tablet_bootstrap.cc:492] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: No bootstrap required, opened a new log
I20260925 05:10:18.872669 5421 ts_tablet_manager.cc:1397] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:18.872813 5421 raft_consensus.cc:348] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.872887 5421 raft_consensus.cc:374] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:18.872910 5421 raft_consensus.cc:729] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Initialized, Role: FOLLOWER
I20260925 05:10:18.872961 5421 consensus_queue.cc:260] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.873063 5421 ts_tablet_manager.cc:1428] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 05:10:18.911307 5311 tablet.cc:2369] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:18.915481 5392 raft_consensus.cc:482] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:18.915601 5392 raft_consensus.cc:504] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.915807 5392 leader_election.cc:288] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781)
I20260925 05:10:18.915972 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb3c280b975437ba6de7a31857a8c67" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" is_pre_election: true
I20260925 05:10:18.916018 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb3c280b975437ba6de7a31857a8c67" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f" is_pre_election: true
I20260925 05:10:18.916098 4984 raft_consensus.cc:2454] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 0.
I20260925 05:10:18.916131 5124 raft_consensus.cc:2454] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 0.
I20260925 05:10:18.916270 5197 leader_election.cc:302] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f567430bcb840f1988d3dd6e7c7d825, a56a2803a7d849a98d2c1c2eb8ee706b; no voters:
I20260925 05:10:18.916375 5392 raft_consensus.cc:2781] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:18.916450 5392 raft_consensus.cc:482] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:18.916483 5392 raft_consensus.cc:3037] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:18.917207 5392 raft_consensus.cc:504] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.917354 5392 leader_election.cc:288] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 election: Requested vote from peers a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903), 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781)
I20260925 05:10:18.917526 4984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb3c280b975437ba6de7a31857a8c67" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b"
I20260925 05:10:18.917619 4984 raft_consensus.cc:3037] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:18.917565 5124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb3c280b975437ba6de7a31857a8c67" candidate_uuid: "5f567430bcb840f1988d3dd6e7c7d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939ffe76074d4448a821b9480fb3db7f"
I20260925 05:10:18.917670 5124 raft_consensus.cc:3037] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:18.918291 5124 raft_consensus.cc:2454] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 1.
I20260925 05:10:18.918401 4984 raft_consensus.cc:2454] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f567430bcb840f1988d3dd6e7c7d825 in term 1.
I20260925 05:10:18.918484 5198 leader_election.cc:302] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f567430bcb840f1988d3dd6e7c7d825, 939ffe76074d4448a821b9480fb3db7f; no voters:
I20260925 05:10:18.918581 5392 raft_consensus.cc:2781] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:18.918629 5392 raft_consensus.cc:686] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 LEADER]: Becoming Leader. State: Replica: 5f567430bcb840f1988d3dd6e7c7d825, State: Running, Role: LEADER
I20260925 05:10:18.918694 5392 consensus_queue.cc:237] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } }
I20260925 05:10:18.919417 4827 catalog_manager.cc:5495] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "5f567430bcb840f1988d3dd6e7c7d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f567430bcb840f1988d3dd6e7c7d825" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20260925 05:10:18.933233 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:18.930744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.3.6.131:42655)
0925 05:10:18.930792 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:18.930794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:18.930805 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:18.930885 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:18.930886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:18.930897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:18.930954 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:18.931043 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.931806 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.932510 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:18.932785 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:18.932861 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:18.932982 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:18.933019 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:18.933085 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:18.933122 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 05:10:18.933148 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:18.935830 4984 raft_consensus.cc:1261] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Refusing update from remote peer 5f567430bcb840f1988d3dd6e7c7d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:18.935861 5124 raft_consensus.cc:1261] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Refusing update from remote peer 5f567430bcb840f1988d3dd6e7c7d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:18.936115 5392 consensus_queue.cc:1035] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56a2803a7d849a98d2c1c2eb8ee706b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:18.936235 5392 consensus_queue.cc:1035] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "939ffe76074d4448a821b9480fb3db7f" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:10:18.941604 5390 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:10:20.462316 4825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:20.462425 4825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:20.463320 4825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f525118edfda4e7e85a3c0045cb6ded8: Sending DeleteTablet for 3 replicas of tablet 1bb3c280b975437ba6de7a31857a8c67
I20260925 05:10:20.463600 5244 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb3c280b975437ba6de7a31857a8c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:20 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58726
I20260925 05:10:20.463619 5104 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb3c280b975437ba6de7a31857a8c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:20 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:57168
I20260925 05:10:20.463622 4964 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb3c280b975437ba6de7a31857a8c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:20 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:37452
I20260925 05:10:20.463778 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 4899
I20260925 05:10:20.463856 5473 tablet_replica.cc:330] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b: stopping tablet replica
I20260925 05:10:20.463922 5473 raft_consensus.cc:2229] T 1bb3c280b975437ba6de7a31857a8c67 P a56a2803a7d849a98d2c1c2eb8ee706b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:20.463861 5472 tablet_replica.cc:330] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: stopping tablet replica
I20260925 05:10:20.465890 5472 raft_consensus.cc:2229] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:20.465981 5472 raft_consensus.cc:2258] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:20.466259 5472 ts_tablet_manager.cc:1905] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:20.468461 5472 ts_tablet_manager.cc:1918] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:10:20.468552 5472 log.cc:1196] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/1bb3c280b975437ba6de7a31857a8c67
I20260925 05:10:20.468770 5471 tablet_replica.cc:330] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: stopping tablet replica
I20260925 05:10:20.468817 5472 ts_tablet_manager.cc:1939] T 1bb3c280b975437ba6de7a31857a8c67 P 5f567430bcb840f1988d3dd6e7c7d825: Deleting consensus metadata
I20260925 05:10:20.468861 5471 raft_consensus.cc:2229] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:20.468919 5471 raft_consensus.cc:2258] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:20.469187 4813 catalog_manager.cc:4841] TS 5f567430bcb840f1988d3dd6e7c7d825 (127.3.6.131:42655): tablet 1bb3c280b975437ba6de7a31857a8c67 (table test-table [id=6db69b8b17994db6bfb5553a8813c93a]) successfully deleted
I20260925 05:10:20.469256 5471 ts_tablet_manager.cc:1905] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:20.470677 5471 ts_tablet_manager.cc:1918] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:10:20.470760 5471 log.cc:1196] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/1bb3c280b975437ba6de7a31857a8c67
I20260925 05:10:20.470978 5471 ts_tablet_manager.cc:1939] T 1bb3c280b975437ba6de7a31857a8c67 P 939ffe76074d4448a821b9480fb3db7f: Deleting consensus metadata
I20260925 05:10:20.471355 4814 catalog_manager.cc:4841] TS 939ffe76074d4448a821b9480fb3db7f (127.3.6.130:46781): tablet 1bb3c280b975437ba6de7a31857a8c67 (table test-table [id=6db69b8b17994db6bfb5553a8813c93a]) successfully deleted
W20260925 05:10:20.471688 4813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:33903 (error 108)
I20260925 05:10:20.472373 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:20.472135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:33903 (local address 127.0.0.1:37466)
0925 05:10:20.472272 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:20.472325 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260925 05:10:20.472384 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 5039
W20260925 05:10:20.472496 4813 catalog_manager.cc:4584] TS a56a2803a7d849a98d2c1c2eb8ee706b (127.3.6.129:33903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bb3c280b975437ba6de7a31857a8c67: Network error: Client connection negotiation failed: client connection to 127.3.6.129:33903: connect: Connection refused (error 111)
I20260925 05:10:20.479825 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 5179
I20260925 05:10:20.485814 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 4795
2026-09-25T05:10:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5666 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:20 dist-test-slave-1lvv krb5kdc[5478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:20 dist-test-slave-1lvv krb5kdc[5478](info): set up 2 sockets
Sep 25 05:10:20 dist-test-slave-1lvv krb5kdc[5478](info): commencing operation
krb5kdc: starting...
W20260925 05:10:21.527745 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:21 dist-test-slave-1lvv krb5kdc[5478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:21.655251 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:38155
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:21.739213 5491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:21.739496 5491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:21.739543 5491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:21.740821 5491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:21.740901 5491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:21.740928 5491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:21.740952 5491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:21.740984 5491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:21.742533 5491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:38155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:21.742846 5491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:21.743103 5491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:10:21.748859 5491 server_base.cc:1034] running on GCE node
W20260925 05:10:21.748944 5497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.749053 5499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.748953 5496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:21.749470 5491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:21.749845 5491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:21.751020 5491 hybrid_clock.cc:648] HybridClock initialized: now 1790313021751006 us; error 50 us; skew 500 ppm
Sep 25 05:10:21 dist-test-slave-1lvv krb5kdc[5478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:21.753687 5491 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:21.754181 5491 webserver.cc:460] Webserver started at http://127.3.6.190:42311/ using document root <none> and password file <none>
I20260925 05:10:21.754381 5491 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:21.754424 5491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:21.754580 5491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:21.755661 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "6141d7fa8a564db5a94f79c264e7e624"
format_stamp: "Formatted at 2026-09-25 05:10:21 on dist-test-slave-1lvv"
server_key: "978f7d94f16d0806593e459d5813b3d8"
server_key_iv: "d4c05524ffad1317c7148f8c2c96a9ae"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.756088 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "6141d7fa8a564db5a94f79c264e7e624"
format_stamp: "Formatted at 2026-09-25 05:10:21 on dist-test-slave-1lvv"
server_key: "978f7d94f16d0806593e459d5813b3d8"
server_key_iv: "d4c05524ffad1317c7148f8c2c96a9ae"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.757562 5491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:21.758337 5506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:21.758564 5491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:21.758671 5491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "6141d7fa8a564db5a94f79c264e7e624"
format_stamp: "Formatted at 2026-09-25 05:10:21 on dist-test-slave-1lvv"
server_key: "978f7d94f16d0806593e459d5813b3d8"
server_key_iv: "d4c05524ffad1317c7148f8c2c96a9ae"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.758738 5491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:21.788511 5491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:21.790757 5491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:21.790898 5491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:21.795388 5491 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:38155
I20260925 05:10:21.795460 5558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:38155 every 8 connection(s)
I20260925 05:10:21.795907 5491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:21.796931 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:21.799525 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624: Bootstrap starting.
I20260925 05:10:21.800267 5559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:21.800608 5559 log.cc:824] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:21.801363 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 5491
I20260925 05:10:21.801455 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:21.801630 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624: No bootstrap required, opened a new log
I20260925 05:10:21.801666 3098 external_mini_cluster.cc:1428] Setting key bda557bedb47222c73146fb7723999f2
I20260925 05:10:21.804450 5559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6141d7fa8a564db5a94f79c264e7e624" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 38155 } }
I20260925 05:10:21.804653 5559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:21.804729 5559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6141d7fa8a564db5a94f79c264e7e624, State: Initialized, Role: FOLLOWER
I20260925 05:10:21.804916 5559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6141d7fa8a564db5a94f79c264e7e624" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 38155 } }
I20260925 05:10:21.805035 5559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:21.805083 5559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:21.805135 5559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:21.805797 5559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6141d7fa8a564db5a94f79c264e7e624" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 38155 } }
I20260925 05:10:21.805928 5559 leader_election.cc:302] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6141d7fa8a564db5a94f79c264e7e624; no voters:
I20260925 05:10:21.806152 5559 leader_election.cc:288] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:21.806185 5564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:21.806313 5564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [term 1 LEADER]: Becoming Leader. State: Replica: 6141d7fa8a564db5a94f79c264e7e624, State: Running, Role: LEADER
I20260925 05:10:21.806401 5564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6141d7fa8a564db5a94f79c264e7e624" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 38155 } }
I20260925 05:10:21.806453 5559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:21.806748 5564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6141d7fa8a564db5a94f79c264e7e624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6141d7fa8a564db5a94f79c264e7e624" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 38155 } } }
I20260925 05:10:21.806824 5564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:21.806965 5564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6141d7fa8a564db5a94f79c264e7e624. Latest consensus state: current_term: 1 leader_uuid: "6141d7fa8a564db5a94f79c264e7e624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6141d7fa8a564db5a94f79c264e7e624" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 38155 } } }
I20260925 05:10:21.807045 5564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624 [sys.catalog]: This master's current role is: LEADER
W20260925 05:10:21.808617 5579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:21.808738 5579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:10:21.808832 5568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:21.809094 5568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:10:21 dist-test-slave-1lvv krb5kdc[5478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:21.810897 5568 catalog_manager.cc:1347] Generated new cluster ID: ab327ee595d64e108d4b74b57bb998a9
I20260925 05:10:21.810958 5568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:21.815253 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:21.802326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.3.6.190:38155)
0925 05:10:21.802608 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:21.802618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:21.802650 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:10:21.803143 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:21.803151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:21.803209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:21.803348 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:21.804317 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:21.805823 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:21.806931 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:21.807505 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:21.807633 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:21.810765 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:21.810769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:21.810770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:21.810784 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:21.812950 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:21.813444 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:21.813447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:21.813447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:21.813575 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:21.813805 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:21.813806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:21.813806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:21.814127 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:21.814781 (+ 654us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:21.814867 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:10:21.815020 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":172,"threads_started":1}
I20260925 05:10:21.819129 5568 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:21.819736 5568 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:21.823992 5568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6141d7fa8a564db5a94f79c264e7e624: Generated new TSK 0
I20260925 05:10:21.824250 5568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:21.859405 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:38155
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:21.948590 5585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:21.948899 5585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:21.948944 5585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:21.950323 5585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:21.950424 5585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:21.950475 5585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:21.952258 5585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:21.952612 5585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:21.953150 5585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:21.959152 5591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.959163 5590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:21.959237 5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:21.959347 5585 server_base.cc:1034] running on GCE node
I20260925 05:10:21.959635 5585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:21.959986 5585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:21.961185 5585 hybrid_clock.cc:648] HybridClock initialized: now 1790313021961169 us; error 48 us; skew 500 ppm
Sep 25 05:10:21 dist-test-slave-1lvv krb5kdc[5478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:21.964177 5585 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:21.964730 5585 webserver.cc:460] Webserver started at http://127.3.6.129:34983/ using document root <none> and password file <none>
I20260925 05:10:21.965022 5585 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:21.965111 5585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:21.965283 5585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:21.966457 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "45347e5965b049f78d8ed581a06e6461"
format_stamp: "Formatted at 2026-09-25 05:10:21 on dist-test-slave-1lvv"
server_key: "5362ec4836c9b553f5985d79b8d7f620"
server_key_iv: "d6703e256855055509281e2f05ef5715"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.966869 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "45347e5965b049f78d8ed581a06e6461"
format_stamp: "Formatted at 2026-09-25 05:10:21 on dist-test-slave-1lvv"
server_key: "5362ec4836c9b553f5985d79b8d7f620"
server_key_iv: "d6703e256855055509281e2f05ef5715"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.968732 5585 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 05:10:21.969924 5600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:21.970152 5585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 05:10:21.970271 5585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "45347e5965b049f78d8ed581a06e6461"
format_stamp: "Formatted at 2026-09-25 05:10:21 on dist-test-slave-1lvv"
server_key: "5362ec4836c9b553f5985d79b8d7f620"
server_key_iv: "d6703e256855055509281e2f05ef5715"
server_key_version: "encryptionkey@0"
I20260925 05:10:21.970368 5585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:21.987531 5585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:21.990067 5585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:21.990281 5585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:21.990597 5585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:21.990954 5585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:21.991010 5585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:21.991048 5585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:21.991079 5585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:21.997406 5585 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:44753
I20260925 05:10:21.997459 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:44753 every 8 connection(s)
I20260925 05:10:21.997808 5585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313021, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:22.006335 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 5585
I20260925 05:10:22.006454 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:22.006685 3098 external_mini_cluster.cc:1428] Setting key 7948c6621ce39f79dfb2775392fddc0a
I20260925 05:10:22.009006 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:21.998617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:41471 (local address 127.3.6.190:38155)
0925 05:10:21.998689 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:21.998691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:21.998922 (+ 231us) server_negotiation.cc:407] Connection header received
0925 05:10:21.999449 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:21.999450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:21.999465 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:21.999577 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:22.000728 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.001261 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.002355 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.002620 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.002690 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:22.005241 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:22.005242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:22.005243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:22.005251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:22.007058 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:22.007697 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:22.007697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:22.007698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:22.007818 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:22.008191 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:22.008192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:22.008192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:22.008320 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:22.008384 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:22.008758 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 05:10:22.008864 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:10:22.009251 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:21.998593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:38155 (local address 127.3.6.129:41471)
0925 05:10:21.998770 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:21.998816 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:21.999321 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:21.999669 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:21.999673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:22.000380 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:22.000592 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:22.000598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.001364 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.001365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:22.002244 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:22.002246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.002728 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.002729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:22.002851 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:22.003501 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:22.003511 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:22.005132 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:22.007158 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:22.007169 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:22.007172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:22.007568 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:22.007883 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:22.007884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:22.007885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:22.008089 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:22.008397 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:22.008404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:22.008598 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 05:10:22.008728 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 05:10:22.008929 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260925 05:10:22.009946 5714 heartbeater.cc:347] Connected to a master server at 127.3.6.190:38155
I20260925 05:10:22.010070 5714 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:22.010298 5714 heartbeater.cc:511] Master 127.3.6.190:38155 requested a full tablet report, sending...
I20260925 05:10:22.010921 5523 ts_manager.cc:194] Registered new tserver with Master: 45347e5965b049f78d8ed581a06e6461 (127.3.6.129:44753)
I20260925 05:10:22.011559 5523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:41471
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:22.035656 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:38155
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:22.120987 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:22.121254 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:22.121282 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:22.122598 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:22.122663 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:22.122694 5719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:22.124315 5719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:22.124605 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:22.124871 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:22.130446 5727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:22.130443 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:22.130443 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:22.130465 5719 server_base.cc:1034] running on GCE node
I20260925 05:10:22.131059 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:22.131350 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:22.132524 5719 hybrid_clock.cc:648] HybridClock initialized: now 1790313022132504 us; error 28 us; skew 500 ppm
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313022, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:22.135468 5719 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:22.136114 5719 webserver.cc:460] Webserver started at http://127.3.6.130:42453/ using document root <none> and password file <none>
I20260925 05:10:22.136399 5719 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:22.136473 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:22.136646 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:22.137807 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "eacaa961d7a04893aa6bda344c5f539c"
format_stamp: "Formatted at 2026-09-25 05:10:22 on dist-test-slave-1lvv"
server_key: "18a92cb3522cbac7ca92d5ff8b819c74"
server_key_iv: "636adec351903e2fee1647660650ea58"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.138198 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "eacaa961d7a04893aa6bda344c5f539c"
format_stamp: "Formatted at 2026-09-25 05:10:22 on dist-test-slave-1lvv"
server_key: "18a92cb3522cbac7ca92d5ff8b819c74"
server_key_iv: "636adec351903e2fee1647660650ea58"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.139698 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:22.140563 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.140751 5719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:22.140866 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "eacaa961d7a04893aa6bda344c5f539c"
format_stamp: "Formatted at 2026-09-25 05:10:22 on dist-test-slave-1lvv"
server_key: "18a92cb3522cbac7ca92d5ff8b819c74"
server_key_iv: "636adec351903e2fee1647660650ea58"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.140947 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:22.156255 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:22.158866 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:22.159000 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:22.159206 5719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:22.159547 5719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:22.159595 5719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.159618 5719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:22.159631 5719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.165863 5719 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:33823
I20260925 05:10:22.165944 5847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:33823 every 8 connection(s)
I20260925 05:10:22.166316 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:22.171746 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 5719
I20260925 05:10:22.171927 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:22.172153 3098 external_mini_cluster.cc:1428] Setting key 32830699780690ede0b8ffd5a1abb65e
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313022, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:22.177851 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.167185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:58581 (local address 127.3.6.190:38155)
0925 05:10:22.167262 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:22.167265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:22.167582 (+ 317us) server_negotiation.cc:407] Connection header received
0925 05:10:22.168236 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:22.168237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:22.168249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:22.168316 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:22.169469 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.169977 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.171212 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.171416 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.171495 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:22.174486 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:22.174488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:22.174488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:22.174499 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:22.176158 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:22.176767 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:22.176769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:22.176769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:22.176838 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:22.177160 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:22.177161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:22.177161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:22.177279 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:22.177333 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:22.177704 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 05:10:22.177781 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:10:22.178202 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.167179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:38155 (local address 127.3.6.130:58581)
0925 05:10:22.167430 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.167472 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:22.168104 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:22.168425 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:22.168429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:22.169142 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:22.169361 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:22.169367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.170113 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.170114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:22.171101 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:22.171104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.171529 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.171530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:22.171690 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:22.172508 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:22.172521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:22.174338 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:22.176284 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:22.176295 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:22.176298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:22.176661 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:22.176918 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:22.176919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:22.176919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:22.177087 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:22.177345 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:22.177352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:22.177544 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 05:10:22.177700 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 05:10:22.177913 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":139,"threads_started":1}
I20260925 05:10:22.178781 5848 heartbeater.cc:347] Connected to a master server at 127.3.6.190:38155
I20260925 05:10:22.178886 5848 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:22.179102 5848 heartbeater.cc:511] Master 127.3.6.190:38155 requested a full tablet report, sending...
I20260925 05:10:22.179570 5523 ts_manager.cc:194] Registered new tserver with Master: eacaa961d7a04893aa6bda344c5f539c (127.3.6.130:33823)
I20260925 05:10:22.180177 5523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:58581
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:22.203647 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:38155
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:22.288024 5853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:22.288260 5853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:22.288287 5853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:22.289549 5853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:22.289599 5853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:22.289628 5853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:22.291137 5853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:22.291406 5853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:22.291656 5853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:22.296942 5861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:22.296963 5859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:22.297084 5853 server_base.cc:1034] running on GCE node
W20260925 05:10:22.296950 5858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:22.297531 5853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:22.297820 5853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:22.299006 5853 hybrid_clock.cc:648] HybridClock initialized: now 1790313022298995 us; error 41 us; skew 500 ppm
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313022, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:22.301853 5853 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:22.302363 5853 webserver.cc:460] Webserver started at http://127.3.6.131:33959/ using document root <none> and password file <none>
I20260925 05:10:22.302634 5853 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:22.302709 5853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:22.302865 5853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:22.304129 5853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "9ab72efd41e24a23bf4a5f1829737c12"
format_stamp: "Formatted at 2026-09-25 05:10:22 on dist-test-slave-1lvv"
server_key: "cb397f5194c41aaf4c7491aa911b5334"
server_key_iv: "a5c9e3288b8bb4781e0aa9afd4a42904"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.304527 5853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "9ab72efd41e24a23bf4a5f1829737c12"
format_stamp: "Formatted at 2026-09-25 05:10:22 on dist-test-slave-1lvv"
server_key: "cb397f5194c41aaf4c7491aa911b5334"
server_key_iv: "a5c9e3288b8bb4781e0aa9afd4a42904"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.305984 5853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:22.306939 5868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.307138 5853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:22.307292 5853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "9ab72efd41e24a23bf4a5f1829737c12"
format_stamp: "Formatted at 2026-09-25 05:10:22 on dist-test-slave-1lvv"
server_key: "cb397f5194c41aaf4c7491aa911b5334"
server_key_iv: "a5c9e3288b8bb4781e0aa9afd4a42904"
server_key_version: "encryptionkey@0"
I20260925 05:10:22.307439 5853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:22.327471 5853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:22.330004 5853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:22.330190 5853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:22.330446 5853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:22.330832 5853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:22.330870 5853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.330915 5853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:22.330943 5853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:22.337082 5853 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:40487
I20260925 05:10:22.337162 5981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:40487 every 8 connection(s)
I20260925 05:10:22.337536 5853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:22.339920 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 5853
I20260925 05:10:22.340024 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:22.340227 3098 external_mini_cluster.cc:1428] Setting key e113557bbeee3085665ebb80bb31791e
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313022, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:22.347702 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.338347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:43001 (local address 127.3.6.190:38155)
0925 05:10:22.338417 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:22.338419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:22.338655 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:10:22.339223 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:22.339225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:22.339239 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:22.339298 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:22.340728 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.341449 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.342496 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.342692 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.342747 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:22.345189 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:22.345190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:22.345191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:22.345199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:22.346450 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:22.346852 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:22.346853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:22.346853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:22.346907 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:22.347141 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:22.347141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:22.347141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:22.347262 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:22.347315 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:22.347560 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 05:10:22.347626 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:10:22.348009 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:22.338343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:38155 (local address 127.3.6.131:43001)
0925 05:10:22.338516 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:22.338556 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:22.339099 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:22.339392 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:22.339396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:22.340276 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:22.340595 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:22.340602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.341542 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.341543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:22.342411 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:22.342414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:22.342770 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:22.342771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:22.342891 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:22.343418 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:22.343427 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:22.345064 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:22.346528 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:22.346536 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:22.346538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:22.346781 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:22.346966 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:22.346966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:22.346966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:22.347084 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:22.347318 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:22.347322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:22.347448 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 05:10:22.347535 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:10:22.347672 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260925 05:10:22.348591 5982 heartbeater.cc:347] Connected to a master server at 127.3.6.190:38155
I20260925 05:10:22.348701 5982 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:22.348869 5982 heartbeater.cc:511] Master 127.3.6.190:38155 requested a full tablet report, sending...
I20260925 05:10:22.349254 5523 ts_manager.cc:194] Registered new tserver with Master: 9ab72efd41e24a23bf4a5f1829737c12 (127.3.6.131:40487)
I20260925 05:10:22.349601 5523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:43001
I20260925 05:10:22.352275 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:10:22.366480 5563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:22.365409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.3.6.190:38155)
0925 05:10:22.365481 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:22.365483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:22.365638 (+ 155us) server_negotiation.cc:407] Connection header received
0925 05:10:22.365689 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:22.365690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:22.365700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:22.365772 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:22.366390 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38782: BlockingRecv error: recv got EOF from 127.0.0.1:38782 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:10:22.367398 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 5585
I20260925 05:10:22.373329 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 5719
I20260925 05:10:22.378206 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 5853
I20260925 05:10:22.382886 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 5491
2026-09-25T05:10:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1896 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5994](info): set up 2 sockets
Sep 25 05:10:22 dist-test-slave-1lvv krb5kdc[5994](info): commencing operation
krb5kdc: starting...
W20260925 05:10:23.427898 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:23 dist-test-slave-1lvv krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:23.555218 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:42779
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:42779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:23.642300 6007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:23.642530 6007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:23.642555 6007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:23.643705 6007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:23.643743 6007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:23.643826 6007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:23.643853 6007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:23.643882 6007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:23.645457 6007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:42779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:42779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:23.645684 6007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:23.645874 6007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:23.651198 6013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:23.651304 6007 server_base.cc:1034] running on GCE node
W20260925 05:10:23.651223 6015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:23.652251 6012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:23.652644 6007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:23.653005 6007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:23.654188 6007 hybrid_clock.cc:648] HybridClock initialized: now 1790313023654161 us; error 35 us; skew 500 ppm
Sep 25 05:10:23 dist-test-slave-1lvv krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313023, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:23.657100 6007 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:23.657644 6007 webserver.cc:460] Webserver started at http://127.3.6.190:36629/ using document root <none> and password file <none>
I20260925 05:10:23.657922 6007 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:23.658001 6007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:23.658172 6007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:23.659394 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "39b49ea0076446309c2608fca385fa1e"
format_stamp: "Formatted at 2026-09-25 05:10:23 on dist-test-slave-1lvv"
server_key: "098ee5690424bfac6870659d5a1de703"
server_key_iv: "130510470956887195c9bdc411399416"
server_key_version: "encryptionkey@0"
I20260925 05:10:23.659898 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "39b49ea0076446309c2608fca385fa1e"
format_stamp: "Formatted at 2026-09-25 05:10:23 on dist-test-slave-1lvv"
server_key: "098ee5690424bfac6870659d5a1de703"
server_key_iv: "130510470956887195c9bdc411399416"
server_key_version: "encryptionkey@0"
I20260925 05:10:23.661526 6007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:23.662386 6022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:23.662622 6007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:23.662743 6007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "39b49ea0076446309c2608fca385fa1e"
format_stamp: "Formatted at 2026-09-25 05:10:23 on dist-test-slave-1lvv"
server_key: "098ee5690424bfac6870659d5a1de703"
server_key_iv: "130510470956887195c9bdc411399416"
server_key_version: "encryptionkey@0"
I20260925 05:10:23.662824 6007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:23.691527 6007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:23.693643 6007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:23.693830 6007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:23.698007 6007 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:42779
I20260925 05:10:23.698045 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:42779 every 8 connection(s)
I20260925 05:10:23.698462 6007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:23.699442 6075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:23.701105 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6007
I20260925 05:10:23.701225 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:23.701404 3098 external_mini_cluster.cc:1428] Setting key 23a4cf432e0e9586425a4fb77037cd29
I20260925 05:10:23.702551 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e: Bootstrap starting.
I20260925 05:10:23.703356 6075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:23.703719 6075 log.cc:824] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:23.704574 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e: No bootstrap required, opened a new log
I20260925 05:10:23.707007 6075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b49ea0076446309c2608fca385fa1e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42779 } }
I20260925 05:10:23.707237 6075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:23.707279 6075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b49ea0076446309c2608fca385fa1e, State: Initialized, Role: FOLLOWER
I20260925 05:10:23.707424 6075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b49ea0076446309c2608fca385fa1e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42779 } }
I20260925 05:10:23.707520 6075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:23.707559 6075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:23.707628 6075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:23.708307 6075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b49ea0076446309c2608fca385fa1e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42779 } }
I20260925 05:10:23.708402 6075 leader_election.cc:302] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39b49ea0076446309c2608fca385fa1e; no voters:
I20260925 05:10:23.708631 6075 leader_election.cc:288] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:23.708680 6080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:23.708899 6080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [term 1 LEADER]: Becoming Leader. State: Replica: 39b49ea0076446309c2608fca385fa1e, State: Running, Role: LEADER
I20260925 05:10:23.708911 6075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:23.709064 6080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b49ea0076446309c2608fca385fa1e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42779 } }
Sep 25 05:10:23 dist-test-slave-1lvv krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:23.709450 6082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39b49ea0076446309c2608fca385fa1e. Latest consensus state: current_term: 1 leader_uuid: "39b49ea0076446309c2608fca385fa1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b49ea0076446309c2608fca385fa1e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42779 } } }
I20260925 05:10:23.709537 6082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:23.709786 6081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39b49ea0076446309c2608fca385fa1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b49ea0076446309c2608fca385fa1e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42779 } } }
I20260925 05:10:23.709920 6081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:23.710108 6086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:23.710680 6086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:23.712296 6086 catalog_manager.cc:1347] Generated new cluster ID: 209d46db749945d0911ece59a382746a
I20260925 05:10:23.712356 6086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:23.720322 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.702007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.3.6.190:42779)
0925 05:10:23.702243 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:23.702254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:23.702292 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:10:23.702839 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:23.702849 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:23.702924 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:23.703117 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:23.703926 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.705330 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.706087 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.706826 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.706941 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:23.709826 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:23.709830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:23.709831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:23.709843 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:23.718320 (+ 8477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:23.718849 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:23.718854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:23.718855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:23.719000 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:23.719240 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:23.719241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:23.719242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:23.719593 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:23.719799 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:23.719887 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 05:10:23.720059 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260925 05:10:23.721372 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:10:23.724913 6086 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:23.725462 6086 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:23.737677 6086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39b49ea0076446309c2608fca385fa1e: Generated new TSK 0
I20260925 05:10:23.737962 6086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:10:23.747136 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:23.743052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.3.6.190:42779)
0925 05:10:23.743112 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:23.743115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:23.743203 (+ 88us) server_negotiation.cc:407] Connection header received
0925 05:10:23.743273 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:23.743274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:23.743285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:23.743349 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:23.743863 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.744416 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.745543 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:23.745763 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:23.745816 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:23.746568 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:23.746570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:23.746570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:23.746827 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:23.746845 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:23.746852 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:23.746899 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:23.746990 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 05:10:23.747022 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:10:23.749840 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6007
2026-09-25T05:10:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1364 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:23 dist-test-slave-1lvv krb5kdc[6108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:23 dist-test-slave-1lvv krb5kdc[6108](info): set up 2 sockets
Sep 25 05:10:23 dist-test-slave-1lvv krb5kdc[6108](info): commencing operation
krb5kdc: starting...
W20260925 05:10:24.788069 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:24 dist-test-slave-1lvv krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:24.915809 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33781
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:33781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:24.999830 6121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:25.000078 6121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:25.000103 6121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:25.001345 6121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:25.001406 6121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:25.001422 6121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:25.001435 6121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:25.001479 6121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:25.003098 6121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:33781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:25.003369 6121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:25.003615 6121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:25.008710 6126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:25.008754 6129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:25.008733 6127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:25.009063 6121 server_base.cc:1034] running on GCE node
I20260925 05:10:25.009305 6121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:25.009552 6121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:25.010706 6121 hybrid_clock.cc:648] HybridClock initialized: now 1790313025010687 us; error 34 us; skew 500 ppm
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:25.013296 6121 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:25.013744 6121 webserver.cc:460] Webserver started at http://127.3.6.190:36649/ using document root <none> and password file <none>
I20260925 05:10:25.013985 6121 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:25.014055 6121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:25.014245 6121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:25.015355 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "475d82c52d004ef9810c116b2e4d5f17"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "c1a04b0b7b9aa64cd606c66d3b025313"
server_key_iv: "8fa99d64cb093899132d4a511f878780"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.015672 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "475d82c52d004ef9810c116b2e4d5f17"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "c1a04b0b7b9aa64cd606c66d3b025313"
server_key_iv: "8fa99d64cb093899132d4a511f878780"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.017040 6121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:25.017771 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.017948 6121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:25.018051 6121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "475d82c52d004ef9810c116b2e4d5f17"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "c1a04b0b7b9aa64cd606c66d3b025313"
server_key_iv: "8fa99d64cb093899132d4a511f878780"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.018139 6121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:25.035148 6121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:25.037323 6121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:25.037520 6121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:25.041726 6121 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:33781
I20260925 05:10:25.041765 6188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:33781 every 8 connection(s)
I20260925 05:10:25.042204 6121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:25.044145 6189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:25.046556 6189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17: Bootstrap starting.
I20260925 05:10:25.047175 6189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:25.047482 6189 log.cc:824] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:25.048280 6189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17: No bootstrap required, opened a new log
I20260925 05:10:25.049887 6189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475d82c52d004ef9810c116b2e4d5f17" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33781 } }
I20260925 05:10:25.050029 6189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:25.050083 6189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475d82c52d004ef9810c116b2e4d5f17, State: Initialized, Role: FOLLOWER
I20260925 05:10:25.050249 6189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475d82c52d004ef9810c116b2e4d5f17" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33781 } }
I20260925 05:10:25.050338 6189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:25.050391 6189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:25.050443 6189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:25.051257 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6121
I20260925 05:10:25.051304 6189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475d82c52d004ef9810c116b2e4d5f17" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33781 } }
I20260925 05:10:25.051378 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:25.051424 6189 leader_election.cc:302] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 475d82c52d004ef9810c116b2e4d5f17; no voters:
I20260925 05:10:25.051573 3098 external_mini_cluster.cc:1428] Setting key eb8a612151b08c66fc2cec4711287939
I20260925 05:10:25.051661 6189 leader_election.cc:288] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:25.051693 6192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:25.051892 6192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [term 1 LEADER]: Becoming Leader. State: Replica: 475d82c52d004ef9810c116b2e4d5f17, State: Running, Role: LEADER
I20260925 05:10:25.051983 6189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:25.052088 6192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475d82c52d004ef9810c116b2e4d5f17" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33781 } }
I20260925 05:10:25.052484 6192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475d82c52d004ef9810c116b2e4d5f17. Latest consensus state: current_term: 1 leader_uuid: "475d82c52d004ef9810c116b2e4d5f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475d82c52d004ef9810c116b2e4d5f17" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33781 } } }
I20260925 05:10:25.052686 6192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:25.052491 6193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "475d82c52d004ef9810c116b2e4d5f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475d82c52d004ef9810c116b2e4d5f17" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33781 } } }
I20260925 05:10:25.052918 6193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:25.052917 6203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:25.053602 6203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:25.055347 6203 catalog_manager.cc:1347] Generated new cluster ID: d091831e3a8849ac8789aab987cff22e
I20260925 05:10:25.055401 6203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:25.064437 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.052146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.3.6.190:33781)
0925 05:10:25.052714 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:25.052725 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:25.052761 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:10:25.052894 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:25.052901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:25.052965 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:25.053116 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:25.054226 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.055608 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.056745 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.057414 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.057540 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.060392 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:25.060398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:25.060398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:25.060413 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:25.062644 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.063051 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.063055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.063055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.063196 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.063406 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.063407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.063407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.063802 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:25.063977 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:25.064054 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 05:10:25.064206 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":68,"threads_started":1}
I20260925 05:10:25.068173 6203 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:25.068933 6203 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:25.079463 6203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 475d82c52d004ef9810c116b2e4d5f17: Generated new TSK 0
I20260925 05:10:25.079747 6203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:25.114238 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33781
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:25.198696 6215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:25.198920 6215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:25.198984 6215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:25.200181 6215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:25.200227 6215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:25.200254 6215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:25.201674 6215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:25.201892 6215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:25.202077 6215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:25.207387 6221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:25.207405 6220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:25.207544 6215 server_base.cc:1034] running on GCE node
W20260925 05:10:25.207393 6223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:25.207952 6215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:25.208231 6215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:25.209393 6215 hybrid_clock.cc:648] HybridClock initialized: now 1790313025209376 us; error 49 us; skew 500 ppm
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:25.212038 6215 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:25.212488 6215 webserver.cc:460] Webserver started at http://127.3.6.129:33171/ using document root <none> and password file <none>
I20260925 05:10:25.212762 6215 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:25.212836 6215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:25.212985 6215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:25.214120 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "f0644b76cdc2449db1485e40708f4590"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "d41c4bf13fe0e9e9f9ea6a974ba4d5a2"
server_key_iv: "6f31acef280fd42543ab2c60b67f4ffc"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.214612 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "f0644b76cdc2449db1485e40708f4590"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "d41c4bf13fe0e9e9f9ea6a974ba4d5a2"
server_key_iv: "6f31acef280fd42543ab2c60b67f4ffc"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.216153 6215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:10:25.217070 6230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.217298 6215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:25.217424 6215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "f0644b76cdc2449db1485e40708f4590"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "d41c4bf13fe0e9e9f9ea6a974ba4d5a2"
server_key_iv: "6f31acef280fd42543ab2c60b67f4ffc"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.217512 6215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:25.242820 6215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:25.244897 6215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:25.245086 6215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:25.245347 6215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:25.245673 6215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:25.245735 6215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.245774 6215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:25.245823 6215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.252118 6215 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:36021
I20260925 05:10:25.252166 6343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:36021 every 8 connection(s)
I20260925 05:10:25.252525 6215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:25.259986 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6215
I20260925 05:10:25.260093 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:25.260313 3098 external_mini_cluster.cc:1428] Setting key fe3661db15cac3c3d3c040bd618eff88
I20260925 05:10:25.262857 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.253335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56419 (local address 127.3.6.190:33781)
0925 05:10:25.253385 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:25.253388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:25.253588 (+ 200us) server_negotiation.cc:407] Connection header received
0925 05:10:25.254228 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:25.254229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:25.254246 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:25.254302 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:25.255318 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.255916 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.256968 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.257203 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.257280 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.259517 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:25.259518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:25.259519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:25.259527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:25.261402 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.261828 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.261829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.261829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.261896 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.262131 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.262132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.262132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.262287 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:25.262357 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:25.262617 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 05:10:25.262717 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:25.262940 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.253293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33781 (local address 127.3.6.129:56419)
0925 05:10:25.253446 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:25.253496 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:25.254090 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:25.254380 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:25.254384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:25.254995 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:25.255208 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:25.255213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.256017 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.256018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:25.256844 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:25.256847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.257291 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.257292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:25.257421 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.257911 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:25.257919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:25.259414 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:25.261488 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:25.261495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:25.261497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:25.261751 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:25.261963 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:25.261964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:25.261964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:25.262065 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:25.262355 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:25.262361 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:25.262482 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 05:10:25.262590 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 05:10:25.262722 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260925 05:10:25.263698 6344 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33781
I20260925 05:10:25.263832 6344 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:25.264043 6344 heartbeater.cc:511] Master 127.3.6.190:33781 requested a full tablet report, sending...
I20260925 05:10:25.264565 6153 ts_manager.cc:194] Registered new tserver with Master: f0644b76cdc2449db1485e40708f4590 (127.3.6.129:36021)
I20260925 05:10:25.265202 6153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:56419
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:25.287930 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33781
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:25.373224 6349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:25.373437 6349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:25.373462 6349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:25.374583 6349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:25.374619 6349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:25.374644 6349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:25.376039 6349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:25.376223 6349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:25.376400 6349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:25.381647 6355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:25.381678 6349 server_base.cc:1034] running on GCE node
W20260925 05:10:25.381850 6357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:25.381718 6354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:25.382160 6349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:25.382402 6349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:25.383550 6349 hybrid_clock.cc:648] HybridClock initialized: now 1790313025383568 us; error 67 us; skew 500 ppm
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:25.386301 6349 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:25.386866 6349 webserver.cc:460] Webserver started at http://127.3.6.130:44755/ using document root <none> and password file <none>
I20260925 05:10:25.387117 6349 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:25.387192 6349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:25.387357 6349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:25.388558 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "0e777be2ca2e4e3ba42dd8839eaa9b55"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "261e821eed7de77396ce4dec5c2890ac"
server_key_iv: "c6ce295341e4c5336d8341b0b31700fa"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.388962 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "0e777be2ca2e4e3ba42dd8839eaa9b55"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "261e821eed7de77396ce4dec5c2890ac"
server_key_iv: "c6ce295341e4c5336d8341b0b31700fa"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.390376 6349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:25.391152 6364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.391341 6349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:25.391455 6349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "0e777be2ca2e4e3ba42dd8839eaa9b55"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "261e821eed7de77396ce4dec5c2890ac"
server_key_iv: "c6ce295341e4c5336d8341b0b31700fa"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.391535 6349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:25.412019 6349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:25.414661 6349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:25.414845 6349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:25.415107 6349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:25.415452 6349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:25.415508 6349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.415556 6349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:25.415596 6349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.422122 6349 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:38785
I20260925 05:10:25.422164 6477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:38785 every 8 connection(s)
I20260925 05:10:25.422533 6349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:25.423862 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6349
I20260925 05:10:25.423952 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:25.424158 3098 external_mini_cluster.cc:1428] Setting key 0c34a834c757cd59bce467c67602ba86
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:25.433812 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.423357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:45527 (local address 127.3.6.190:33781)
0925 05:10:25.423453 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:25.423455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:25.423643 (+ 188us) server_negotiation.cc:407] Connection header received
0925 05:10:25.424316 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:25.424318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:25.424331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:25.424383 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:25.425482 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.425993 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.427538 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.427743 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.427873 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.430391 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:25.430394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:25.430394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:25.430405 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:25.432222 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.432714 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.432715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.432716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.432777 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.433074 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.433075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.433075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.433236 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:25.433316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:25.433588 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 05:10:25.433700 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:10:25.433914 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.423317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33781 (local address 127.3.6.130:45527)
0925 05:10:25.423507 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:25.423548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:25.424202 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:25.424457 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:25.424461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:25.425170 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:25.425376 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:25.425382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.426096 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.426097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:25.427436 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:25.427440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.427888 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.427889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:25.428049 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.428603 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:25.428614 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:25.430285 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:25.432327 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:25.432335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:25.432337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:25.432623 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:25.432872 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:25.432873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:25.432874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:25.432995 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:25.433312 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:25.433316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:25.433446 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 05:10:25.433533 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:10:25.433695 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
I20260925 05:10:25.434460 6478 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33781
I20260925 05:10:25.434569 6478 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:25.434731 6478 heartbeater.cc:511] Master 127.3.6.190:33781 requested a full tablet report, sending...
I20260925 05:10:25.435117 6153 ts_manager.cc:194] Registered new tserver with Master: 0e777be2ca2e4e3ba42dd8839eaa9b55 (127.3.6.130:38785)
I20260925 05:10:25.435561 6153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:45527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:25.455114 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33781
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:25.538802 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:25.539033 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:25.539058 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:25.540374 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:25.540424 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:25.540453 6483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:25.541886 6483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:25.542083 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:25.542275 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:25.547375 6489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:25.547492 6483 server_base.cc:1034] running on GCE node
W20260925 05:10:25.547392 6491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:25.547392 6488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:25.547891 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:25.548163 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:25.549337 6483 hybrid_clock.cc:648] HybridClock initialized: now 1790313025549327 us; error 34 us; skew 500 ppm
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:25.552073 6483 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:25.552690 6483 webserver.cc:460] Webserver started at http://127.3.6.131:35423/ using document root <none> and password file <none>
I20260925 05:10:25.552973 6483 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:25.553023 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:25.553185 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:25.554350 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "8a91212899784faca50b2c4b24db21d6"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "b055158bef48b80fb0701b7f6c299429"
server_key_iv: "815b9e97e7292421e949c1853616a909"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.554797 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "8a91212899784faca50b2c4b24db21d6"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "b055158bef48b80fb0701b7f6c299429"
server_key_iv: "815b9e97e7292421e949c1853616a909"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.556262 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:25.557039 6498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.557251 6483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:25.557361 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "8a91212899784faca50b2c4b24db21d6"
format_stamp: "Formatted at 2026-09-25 05:10:25 on dist-test-slave-1lvv"
server_key: "b055158bef48b80fb0701b7f6c299429"
server_key_iv: "815b9e97e7292421e949c1853616a909"
server_key_version: "encryptionkey@0"
I20260925 05:10:25.557439 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:25.574695 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:25.576715 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:25.576886 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:25.577135 6483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:25.577458 6483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:25.577512 6483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.577559 6483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:25.577585 6483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:25.583899 6483 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:36419
I20260925 05:10:25.583941 6611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:36419 every 8 connection(s)
I20260925 05:10:25.584319 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:25.591053 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6483
I20260925 05:10:25.591161 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:25.591341 3098 external_mini_cluster.cc:1428] Setting key 9a7f3fa1c56292259a5a31554603be03
I20260925 05:10:25.593900 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.585156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:46961 (local address 127.3.6.190:33781)
0925 05:10:25.585227 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:25.585229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:25.585485 (+ 256us) server_negotiation.cc:407] Connection header received
0925 05:10:25.586002 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:25.586004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:25.586015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:25.586087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:25.587064 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.587591 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.588679 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.588866 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.588926 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.591101 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:25.591103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:25.591103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:25.591110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:25.592747 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.593125 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.593126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.593126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.593186 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.593399 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.593399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.593399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.593501 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:25.593551 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:25.593774 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:10:25.593833 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:10:25.594106 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.585150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33781 (local address 127.3.6.131:46961)
0925 05:10:25.585320 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:25.585365 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:25.585878 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:25.586147 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:25.586150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:25.586754 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:25.586961 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:25.586967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.587678 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.587679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:25.588595 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:25.588598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.588945 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.588946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:25.589047 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.589549 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:25.589557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:25.590998 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:25.592819 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:25.592825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:25.592827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:25.593058 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:25.593243 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:25.593243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:25.593243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:25.593343 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:25.593554 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:25.593558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:25.593680 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 05:10:25.593762 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:10:25.593881 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260925 05:10:25.594650 6612 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33781
I20260925 05:10:25.594738 6612 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:25.594882 6612 heartbeater.cc:511] Master 127.3.6.190:33781 requested a full tablet report, sending...
I20260925 05:10:25.595244 6153 ts_manager.cc:194] Registered new tserver with Master: 8a91212899784faca50b2c4b24db21d6 (127.3.6.131:36419)
I20260925 05:10:25.595649 6153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:46961
I20260925 05:10:25.603274 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:25.627235 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:25.619714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.3.6.190:33781)
0925 05:10:25.619812 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:25.619814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:25.619889 (+ 75us) server_negotiation.cc:407] Connection header received
0925 05:10:25.619940 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:25.619941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:25.619951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:25.620058 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:25.620670 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.621283 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.622040 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:25.622231 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:25.622311 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:25.624705 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:25.624707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:25.624707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:25.624717 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:25.625985 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.626356 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.626358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.626358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.626419 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:25.626628 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:25.626629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:25.626629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:25.626788 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:25.626900 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:25.627019 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 05:10:25.627107 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 05:10:25.627482 6153 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60550
I20260925 05:10:25.627941 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6215
I20260925 05:10:25.634768 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6349
I20260925 05:10:25.641661 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6483
I20260925 05:10:25.646544 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6121
2026-09-25T05:10:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1898 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6624](info): set up 2 sockets
Sep 25 05:10:25 dist-test-slave-1lvv krb5kdc[6624](info): commencing operation
krb5kdc: starting...
W20260925 05:10:26.684100 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:26 dist-test-slave-1lvv krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:26.813028 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44631
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:26.896349 6637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:26.896579 6637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:26.896603 6637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:26.897780 6637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:26.897825 6637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:26.897838 6637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:26.897851 6637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:26.897864 6637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:26.899438 6637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:26.899734 6637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:26.900002 6637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:26.905055 6645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:26.905066 6643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:26.905081 6642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:26.905390 6637 server_base.cc:1034] running on GCE node
I20260925 05:10:26.905668 6637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:26.906023 6637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:26.907236 6637 hybrid_clock.cc:648] HybridClock initialized: now 1790313026907213 us; error 41 us; skew 500 ppm
Sep 25 05:10:26 dist-test-slave-1lvv krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313026, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:26.910065 6637 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:26.910589 6637 webserver.cc:460] Webserver started at http://127.3.6.190:37731/ using document root <none> and password file <none>
I20260925 05:10:26.910838 6637 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:26.910907 6637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:26.911065 6637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:26.912231 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "fca50b2eaeef4448936be79fa8406e80"
format_stamp: "Formatted at 2026-09-25 05:10:26 on dist-test-slave-1lvv"
server_key: "befcaafd54c2ad1dd169efa9c2b1621e"
server_key_iv: "187db3002944a8548a5b3e0dfa7027d6"
server_key_version: "encryptionkey@0"
I20260925 05:10:26.912608 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "fca50b2eaeef4448936be79fa8406e80"
format_stamp: "Formatted at 2026-09-25 05:10:26 on dist-test-slave-1lvv"
server_key: "befcaafd54c2ad1dd169efa9c2b1621e"
server_key_iv: "187db3002944a8548a5b3e0dfa7027d6"
server_key_version: "encryptionkey@0"
I20260925 05:10:26.914022 6637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:26.914852 6652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:26.915076 6637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:26.915190 6637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "fca50b2eaeef4448936be79fa8406e80"
format_stamp: "Formatted at 2026-09-25 05:10:26 on dist-test-slave-1lvv"
server_key: "befcaafd54c2ad1dd169efa9c2b1621e"
server_key_iv: "187db3002944a8548a5b3e0dfa7027d6"
server_key_version: "encryptionkey@0"
I20260925 05:10:26.915269 6637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:26.925789 6637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:26.928072 6637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:26.928249 6637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:26.932582 6637 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44631
I20260925 05:10:26.932619 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44631 every 8 connection(s)
I20260925 05:10:26.933156 6637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:26.933887 6705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:26.936317 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80: Bootstrap starting.
I20260925 05:10:26.936914 6705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:26.937167 6705 log.cc:824] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:26.937916 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80: No bootstrap required, opened a new log
I20260925 05:10:26.938755 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6637
I20260925 05:10:26.938901 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:26.939121 3098 external_mini_cluster.cc:1428] Setting key 94d680d77ee88737fb43c583e89b4834
I20260925 05:10:26.939814 6705 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca50b2eaeef4448936be79fa8406e80" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44631 } }
I20260925 05:10:26.940003 6705 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:26.940032 6705 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca50b2eaeef4448936be79fa8406e80, State: Initialized, Role: FOLLOWER
I20260925 05:10:26.940184 6705 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca50b2eaeef4448936be79fa8406e80" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44631 } }
I20260925 05:10:26.940263 6705 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:26.940286 6705 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:26.940356 6705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:26.941071 6705 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca50b2eaeef4448936be79fa8406e80" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44631 } }
I20260925 05:10:26.941201 6705 leader_election.cc:302] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca50b2eaeef4448936be79fa8406e80; no voters:
I20260925 05:10:26.941386 6705 leader_election.cc:288] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:26.941458 6710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:26.941663 6710 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [term 1 LEADER]: Becoming Leader. State: Replica: fca50b2eaeef4448936be79fa8406e80, State: Running, Role: LEADER
I20260925 05:10:26.941679 6705 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:26.941851 6710 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca50b2eaeef4448936be79fa8406e80" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44631 } }
I20260925 05:10:26.942354 6711 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca50b2eaeef4448936be79fa8406e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca50b2eaeef4448936be79fa8406e80" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44631 } } }
I20260925 05:10:26.942488 6711 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:26.942348 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca50b2eaeef4448936be79fa8406e80. Latest consensus state: current_term: 1 leader_uuid: "fca50b2eaeef4448936be79fa8406e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca50b2eaeef4448936be79fa8406e80" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44631 } } }
I20260925 05:10:26.942749 6719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:26.942847 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:26.943377 6719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:26.946180 6719 catalog_manager.cc:1347] Generated new cluster ID: f43f6f5cd8434342b09c701e15640630
I20260925 05:10:26.946242 6719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:10:26 dist-test-slave-1lvv krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:26.952386 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:26.939735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.3.6.190:44631)
0925 05:10:26.940500 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:26.940509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:26.940536 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:10:26.940651 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:26.940657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:26.940704 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:26.940824 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:26.941698 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:26.943353 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:26.944671 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:26.945143 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:26.945237 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:26.948255 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:26.948260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:26.948260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:26.948274 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:26.950452 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:26.950934 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:26.950937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:26.950938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:26.951086 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:26.951335 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:26.951335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:26.951336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:26.951662 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:26.951844 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:26.951982 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:10:26.952136 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":677,"threads_started":1}
I20260925 05:10:26.965575 6719 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:26.966377 6719 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:26.972296 6719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca50b2eaeef4448936be79fa8406e80: Generated new TSK 0
I20260925 05:10:26.972568 6719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:27.003573 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44631
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:27.090121 6731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:27.090345 6731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:27.090370 6731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:27.091512 6731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:27.091555 6731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:27.091581 6731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:27.093114 6731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:27.093303 6731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:27.093502 6731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:27.098811 6739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:27.098785 6737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:27.099001 6731 server_base.cc:1034] running on GCE node
W20260925 05:10:27.098785 6736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:27.099262 6731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:27.099853 6731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:27.101033 6731 hybrid_clock.cc:648] HybridClock initialized: now 1790313027101005 us; error 45 us; skew 500 ppm
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:27.103802 6731 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:27.104360 6731 webserver.cc:460] Webserver started at http://127.3.6.129:45887/ using document root <none> and password file <none>
I20260925 05:10:27.104651 6731 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:27.104732 6731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:27.104902 6731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:27.106212 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "4b5e5b5e2bab4b8b89f5fdb2f379af82"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "ffa06f3ffb8f1db87f79c43a7b8e8bf8"
server_key_iv: "ffb5c402129d9f062122a62851666e47"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.106632 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "4b5e5b5e2bab4b8b89f5fdb2f379af82"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "ffa06f3ffb8f1db87f79c43a7b8e8bf8"
server_key_iv: "ffb5c402129d9f062122a62851666e47"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.108174 6731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 05:10:27.108992 6746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.109200 6731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.109277 6731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "4b5e5b5e2bab4b8b89f5fdb2f379af82"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "ffa06f3ffb8f1db87f79c43a7b8e8bf8"
server_key_iv: "ffb5c402129d9f062122a62851666e47"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.109370 6731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:27.130012 6731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:27.132465 6731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:27.132663 6731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:27.132964 6731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:27.133353 6731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:27.133416 6731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.133456 6731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:27.133488 6731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.140134 6731 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:40881
I20260925 05:10:27.140177 6859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:40881 every 8 connection(s)
I20260925 05:10:27.140642 6731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:27.149919 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6731
I20260925 05:10:27.150015 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:27.150211 3098 external_mini_cluster.cc:1428] Setting key d58a4515d1a537925553ee1051a4a1d2
I20260925 05:10:27.150626 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.141522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:52245 (local address 127.3.6.190:44631)
0925 05:10:27.141629 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:27.141632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:27.141873 (+ 241us) server_negotiation.cc:407] Connection header received
0925 05:10:27.143015 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:27.143016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:27.143031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:27.143094 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:27.143873 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.144405 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.145411 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.145608 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.145660 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.147839 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:27.147840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:27.147840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:27.147849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:27.149153 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.149535 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.149536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.149536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.149587 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.149882 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.149882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.149882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.150022 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:27.150086 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:27.150443 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 05:10:27.150534 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:10:27.150835 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.141465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44631 (local address 127.3.6.129:52245)
0925 05:10:27.141720 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:27.141781 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:27.142909 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:27.143180 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:27.143184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:27.143460 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:27.143731 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:27.143736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.144494 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.144495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:27.145335 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:27.145337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.145684 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.145684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:27.145785 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.146279 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:27.146287 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:27.147691 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:27.149234 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:27.149240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:27.149242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:27.149469 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:27.149645 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:27.149645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:27.149646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:27.149807 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:27.150099 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:27.150108 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:27.150273 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 05:10:27.150407 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 05:10:27.150581 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
I20260925 05:10:27.151371 6860 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44631
I20260925 05:10:27.151481 6860 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:27.151729 6860 heartbeater.cc:511] Master 127.3.6.190:44631 requested a full tablet report, sending...
I20260925 05:10:27.152382 6669 ts_manager.cc:194] Registered new tserver with Master: 4b5e5b5e2bab4b8b89f5fdb2f379af82 (127.3.6.129:40881)
I20260925 05:10:27.153164 6669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:52245
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:27.178787 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44631
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:27.262565 6865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:27.262784 6865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:27.262807 6865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:27.263937 6865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:27.263974 6865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:27.264000 6865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:27.265460 6865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:27.265666 6865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:27.265854 6865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:27.271126 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:27.271130 6871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:27.271126 6873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:27.271584 6865 server_base.cc:1034] running on GCE node
I20260925 05:10:27.271884 6865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:27.272187 6865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:27.273355 6865 hybrid_clock.cc:648] HybridClock initialized: now 1790313027273343 us; error 45 us; skew 500 ppm
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:27.276062 6865 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:27.276571 6865 webserver.cc:460] Webserver started at http://127.3.6.130:46369/ using document root <none> and password file <none>
I20260925 05:10:27.276825 6865 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:27.276893 6865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:27.277043 6865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:27.278154 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "97a2d20dbbbc44c99bdf1812ef22dba5"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "1f4776f78934d03c299dc34572d709f9"
server_key_iv: "4c4afc96ab255fdee53b79eb3a989bb0"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.278532 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "97a2d20dbbbc44c99bdf1812ef22dba5"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "1f4776f78934d03c299dc34572d709f9"
server_key_iv: "4c4afc96ab255fdee53b79eb3a989bb0"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.280036 6865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:27.280808 6880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.280977 6865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:27.281087 6865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "97a2d20dbbbc44c99bdf1812ef22dba5"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "1f4776f78934d03c299dc34572d709f9"
server_key_iv: "4c4afc96ab255fdee53b79eb3a989bb0"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.281163 6865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:27.294476 6865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:27.296943 6865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:27.297123 6865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:27.297375 6865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:27.297710 6865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:27.297778 6865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.297820 6865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:27.297865 6865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.304179 6865 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:44755
I20260925 05:10:27.304224 6993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:44755 every 8 connection(s)
I20260925 05:10:27.304670 6865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:27.314337 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.305489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:45463 (local address 127.3.6.190:44631)
0925 05:10:27.305549 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:27.305551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:27.305744 (+ 193us) server_negotiation.cc:407] Connection header received
0925 05:10:27.306750 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:27.306752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:27.306762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:27.306828 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:27.307454 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.308039 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.309093 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.309283 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.309335 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.311566 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:27.311568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:27.311568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:27.311575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:27.312956 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.313335 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.313336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.313336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.313389 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.313668 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.313669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.313670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.313789 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:27.313868 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:27.314149 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 05:10:27.314244 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:10:27.314523 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.305443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44631 (local address 127.3.6.130:45463)
0925 05:10:27.305613 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:27.305656 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:27.306646 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:27.306883 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:27.306887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:27.307157 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:27.307377 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:27.307381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.308151 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.308152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:27.309013 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:27.309015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.309377 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.309378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:27.309501 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.310079 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:27.310088 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:27.311475 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:27.313025 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:27.313030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:27.313032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:27.313266 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:27.313468 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:27.313469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:27.313470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:27.313589 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:27.313861 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:27.313867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:27.314003 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 05:10:27.314114 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:10:27.314278 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260925 05:10:27.314725 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6865
I20260925 05:10:27.314848 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:27.315045 3098 external_mini_cluster.cc:1428] Setting key 356d5cdda31efa1603b7e96f58fd23d3
I20260925 05:10:27.315021 6994 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44631
I20260925 05:10:27.315140 6994 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:27.315359 6994 heartbeater.cc:511] Master 127.3.6.190:44631 requested a full tablet report, sending...
I20260925 05:10:27.315793 6669 ts_manager.cc:194] Registered new tserver with Master: 97a2d20dbbbc44c99bdf1812ef22dba5 (127.3.6.130:44755)
I20260925 05:10:27.316161 6669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:45463
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:27.343036 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44631
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:27.429692 6999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:27.429960 6999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:27.430003 6999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:27.431216 6999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:27.431283 6999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:27.431331 6999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:27.432973 6999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:27.433261 6999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:27.433501 6999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:27.438820 7004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:27.438866 6999 server_base.cc:1034] running on GCE node
W20260925 05:10:27.438906 7005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:27.438820 7007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:27.439280 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:27.439548 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:27.440722 6999 hybrid_clock.cc:648] HybridClock initialized: now 1790313027440650 us; error 89 us; skew 500 ppm
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:27.443907 6999 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:27.444427 6999 webserver.cc:460] Webserver started at http://127.3.6.131:37201/ using document root <none> and password file <none>
I20260925 05:10:27.444689 6999 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:27.444765 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:27.444931 6999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:27.446069 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "fc3a2f3bf6d443138a578df02e3e01df"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "3d19007741bea951bdbf6eb4395faf81"
server_key_iv: "004af92269fe33f9a498ba1132a0aac1"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.446480 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "fc3a2f3bf6d443138a578df02e3e01df"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "3d19007741bea951bdbf6eb4395faf81"
server_key_iv: "004af92269fe33f9a498ba1132a0aac1"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.448074 6999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:27.448890 7014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.449069 6999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:27.449183 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "fc3a2f3bf6d443138a578df02e3e01df"
format_stamp: "Formatted at 2026-09-25 05:10:27 on dist-test-slave-1lvv"
server_key: "3d19007741bea951bdbf6eb4395faf81"
server_key_iv: "004af92269fe33f9a498ba1132a0aac1"
server_key_version: "encryptionkey@0"
I20260925 05:10:27.449261 6999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:27.461247 6999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:27.463994 6999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:27.464178 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:27.464427 6999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:27.464777 6999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:27.464843 6999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.464892 6999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:27.464927 6999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:27.471909 6999 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:34795
I20260925 05:10:27.471982 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:34795 every 8 connection(s)
I20260925 05:10:27.472390 6999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:27.479054 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 6999
I20260925 05:10:27.479172 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:27.479391 3098 external_mini_cluster.cc:1428] Setting key 17332a5d6b94837b9795449e137585ab
I20260925 05:10:27.482635 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.473515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:45183 (local address 127.3.6.190:44631)
0925 05:10:27.473607 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:27.473610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:27.473847 (+ 237us) server_negotiation.cc:407] Connection header received
0925 05:10:27.474797 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:27.474799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:27.474812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:27.474888 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:27.475496 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.476119 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.477163 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.477352 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.477409 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.479659 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:27.479661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:27.479661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:27.479671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:27.481282 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.481682 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.481683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.481684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.481745 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.481981 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.481981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.481982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.482151 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:27.482232 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:27.482493 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 05:10:27.482565 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:10:27.482828 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.473488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44631 (local address 127.3.6.131:45183)
0925 05:10:27.473707 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:27.473745 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:27.474663 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:27.474948 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:27.474951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:27.475223 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:27.475420 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:27.475424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.476200 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.476201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:27.477083 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:27.477086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.477427 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.477428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:27.477525 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.478025 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:27.478034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:27.479511 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:27.481366 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:27.481373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:27.481375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:27.481604 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:27.481811 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:27.481812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:27.481812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:27.481913 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:27.482227 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:27.482232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:27.482356 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 05:10:27.482451 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:10:27.482578 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260925 05:10:27.483410 7128 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44631
I20260925 05:10:27.483500 7128 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:27.483649 7128 heartbeater.cc:511] Master 127.3.6.190:44631 requested a full tablet report, sending...
I20260925 05:10:27.484135 6669 ts_manager.cc:194] Registered new tserver with Master: fc3a2f3bf6d443138a578df02e3e01df (127.3.6.131:34795)
I20260925 05:10:27.484560 6669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:45183
I20260925 05:10:27.491542 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:27.516299 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.508161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.3.6.129:40881)
0925 05:10:27.508339 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:27.508346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:27.508360 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:10:27.508402 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:27.508404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:27.508420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:27.508507 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:27.509069 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.509896 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.510766 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.511132 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.511202 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.513553 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:27.513565 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:27.513565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:27.513571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:27.515233 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.515536 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.515538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.515539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.515609 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.515840 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.515840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.515841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.515974 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:27.516093 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:27.516150 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 05:10:27.516213 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":129,"threads_started":1}
Sep 25 05:10:27 dist-test-slave-1lvv krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:27.524230 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:27.517896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.3.6.190:44631)
0925 05:10:27.517942 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:27.517944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:27.517956 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:27.518019 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:27.518020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:27.518029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:27.518079 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:27.518509 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.519007 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.519667 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:27.519903 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:27.519955 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:27.522085 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:27.522086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:27.522086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:27.522093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:27.523210 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.523513 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.523514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.523514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.523562 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:27.523725 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:27.523725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:27.523726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:27.523920 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:27.523997 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:27.524103 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 05:10:27.524161 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 05:10:27.524463 6669 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37980
I20260925 05:10:27.524828 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6731
I20260925 05:10:27.530182 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6865
I20260925 05:10:27.534667 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6999
I20260925 05:10:27.539229 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 6637
2026-09-25T05:10:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1894 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:28 dist-test-slave-1lvv krb5kdc[7140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:28 dist-test-slave-1lvv krb5kdc[7140](info): set up 2 sockets
Sep 25 05:10:28 dist-test-slave-1lvv krb5kdc[7140](info): commencing operation
krb5kdc: starting...
W20260925 05:10:30.006613 3098 mini_kdc.cc:121] Time spent starting KDC: real 2.444s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 05:10:30.259893 3098 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.253s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 05:10:30.526630 3098 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.267s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 05:10:30.739197 3098 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.212s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/test-user.keytab.
W20260925 05:10:31.532629 3098 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.793s user 0.000s sys 0.002s
Sep 25 05:10:31 dist-test-slave-1lvv krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
W20260925 05:10:31.990690 3098 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.6.190: real 0.434s user 0.001s sys 0.002s
I20260925 05:10:31.991068 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44783
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:32.081045 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:32.081290 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:32.081315 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:32.082538 7153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:32.082588 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:32.082603 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:32.082616 7153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:32.082629 7153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:32.084227 7153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:32.084517 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:32.084780 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:32.090065 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.090044 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:32.090282 7153 server_base.cc:1034] running on GCE node
W20260925 05:10:32.090046 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:32.090595 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:32.090905 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:32.092051 7153 hybrid_clock.cc:648] HybridClock initialized: now 1790313032092029 us; error 39 us; skew 500 ppm
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:32.094587 7153 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:32.094990 7153 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:32.095043 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:32.095216 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:32.096444 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "82d01b84634d4ddba071b8ba1aa4daca"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "9061df7e908a32fffac6d9f2bd5acf02"
server_key_iv: "1c5394a6042ec07c914f5c05e7323ee2"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.096895 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "82d01b84634d4ddba071b8ba1aa4daca"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "9061df7e908a32fffac6d9f2bd5acf02"
server_key_iv: "1c5394a6042ec07c914f5c05e7323ee2"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.098292 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:32.099048 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.099253 7153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:32.099347 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "82d01b84634d4ddba071b8ba1aa4daca"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "9061df7e908a32fffac6d9f2bd5acf02"
server_key_iv: "1c5394a6042ec07c914f5c05e7323ee2"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.099422 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:32.119618 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:32.121794 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:32.121945 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:32.126397 7153 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44783
I20260925 05:10:32.126426 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44783 every 8 connection(s)
I20260925 05:10:32.126757 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:32.127091 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 7153
I20260925 05:10:32.127197 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:32.127417 3098 external_mini_cluster.cc:1428] Setting key ba4bf554baa018d5d0ecf3d89770e528
I20260925 05:10:32.128103 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:32.131808 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca: Bootstrap starting.
I20260925 05:10:32.132546 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:32.132860 7220 log.cc:824] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:32.133760 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca: No bootstrap required, opened a new log
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:32.136157 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d01b84634d4ddba071b8ba1aa4daca" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44783 } }
I20260925 05:10:32.136329 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:32.136355 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d01b84634d4ddba071b8ba1aa4daca, State: Initialized, Role: FOLLOWER
I20260925 05:10:32.136507 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d01b84634d4ddba071b8ba1aa4daca" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44783 } }
I20260925 05:10:32.136595 7220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:32.136634 7220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:32.136693 7220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:32.137476 7220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d01b84634d4ddba071b8ba1aa4daca" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44783 } }
I20260925 05:10:32.137590 7220 leader_election.cc:302] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d01b84634d4ddba071b8ba1aa4daca; no voters:
I20260925 05:10:32.137756 7220 leader_election.cc:288] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:32.137814 7225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:32.138020 7225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [term 1 LEADER]: Becoming Leader. State: Replica: 82d01b84634d4ddba071b8ba1aa4daca, State: Running, Role: LEADER
I20260925 05:10:32.138048 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:32.138115 7225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d01b84634d4ddba071b8ba1aa4daca" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44783 } }
I20260925 05:10:32.138510 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d01b84634d4ddba071b8ba1aa4daca. Latest consensus state: current_term: 1 leader_uuid: "82d01b84634d4ddba071b8ba1aa4daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d01b84634d4ddba071b8ba1aa4daca" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44783 } } }
I20260925 05:10:32.138543 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d01b84634d4ddba071b8ba1aa4daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d01b84634d4ddba071b8ba1aa4daca" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44783 } } }
I20260925 05:10:32.138669 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:32.138693 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:32.138893 7234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:32.139669 7234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:32.140445 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.128038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.3.6.190:44783)
0925 05:10:32.128250 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.128260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.128295 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:10:32.129205 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.129216 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.129296 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.129473 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:32.130078 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.131487 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.132374 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.133027 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.133140 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.135889 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:32.135894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:32.135894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:32.135910 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:32.138081 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.138563 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.138566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.138566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.138693 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.138946 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.138947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.138947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.139295 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:32.140085 (+ 790us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.140157 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.140264 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260925 05:10:32.141352 7234 catalog_manager.cc:1347] Generated new cluster ID: d84793c04aaf4ff6bc0b6fc2a234f601
I20260925 05:10:32.141407 7234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:32.156213 7234 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:32.156831 7234 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:32.162581 7234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca: Generated new TSK 0
I20260925 05:10:32.162814 7234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:32.199460 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44783
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:32.290680 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:32.290894 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:32.290920 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:32.292120 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:32.292163 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:32.292196 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:32.293665 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:32.293854 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:32.294030 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:32.299204 7254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.299206 7252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:32.299263 7246 server_base.cc:1034] running on GCE node
W20260925 05:10:32.299234 7251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:32.299654 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:32.299937 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:32.301105 7246 hybrid_clock.cc:648] HybridClock initialized: now 1790313032301068 us; error 55 us; skew 500 ppm
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:32.304044 7246 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:32.304416 7246 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:32.304466 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:32.304569 7246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:32.305698 7246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "c765d3cda56742edaaed9a888b00a5ba"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "e9757f0ef366665269d447d742907fb9"
server_key_iv: "8d2f1d60dc4d59fc4f3c4a0490137754"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.306025 7246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "c765d3cda56742edaaed9a888b00a5ba"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "e9757f0ef366665269d447d742907fb9"
server_key_iv: "8d2f1d60dc4d59fc4f3c4a0490137754"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.307430 7246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:32.308254 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.308425 7246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.308549 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "c765d3cda56742edaaed9a888b00a5ba"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "e9757f0ef366665269d447d742907fb9"
server_key_iv: "8d2f1d60dc4d59fc4f3c4a0490137754"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.308631 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:32.320261 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:32.322371 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:32.322567 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:32.322832 7246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:32.323170 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:32.323227 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.323269 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:32.323312 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.329552 7246 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:42939
I20260925 05:10:32.329597 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:42939 every 8 connection(s)
I20260925 05:10:32.329928 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:32.336229 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 7246
I20260925 05:10:32.336372 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:32.336628 3098 external_mini_cluster.cc:1428] Setting key c35f5524d94c4c7843fe6dfd68ba5593
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:32.341893 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.330783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44783 (local address 127.3.6.129:60747)
0925 05:10:32.330977 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.331030 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.331526 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.331831 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.331835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.332157 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:32.332704 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.332710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.333495 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.333495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.334389 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.334391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.334867 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.334868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.335002 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.335650 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:32.335661 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:32.337743 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:32.340217 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:32.340227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:32.340229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:32.340541 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:32.340819 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:32.340820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:32.340821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:32.340957 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:32.341295 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:32.341302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:32.341447 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.341544 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.341705 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260925 05:10:32.341902 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.330819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:60747 (local address 127.3.6.190:44783)
0925 05:10:32.330891 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.330893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.331130 (+ 237us) server_negotiation.cc:407] Connection header received
0925 05:10:32.331617 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.331618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.331628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.331697 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:32.332824 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.333391 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.334510 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.334750 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.334822 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.337860 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:32.337861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:32.337862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:32.337872 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:32.340069 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.340636 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.340638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.340638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.340715 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.341038 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.341039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.341039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.341217 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:32.341298 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.341601 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.341735 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:10:32.342916 7374 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44783
I20260925 05:10:32.343052 7374 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:32.343279 7374 heartbeater.cc:511] Master 127.3.6.190:44783 requested a full tablet report, sending...
I20260925 05:10:32.344048 7184 ts_manager.cc:194] Registered new tserver with Master: c765d3cda56742edaaed9a888b00a5ba (127.3.6.129:42939)
I20260925 05:10:32.344779 7184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:60747
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:32.368109 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44783
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:32.457165 7379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:32.457455 7379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:32.457509 7379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:32.458734 7379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:32.458811 7379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:32.458863 7379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:32.460444 7379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:32.460734 7379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:32.460984 7379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:10:32.466257 7379 server_base.cc:1034] running on GCE node
W20260925 05:10:32.466327 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.466287 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.466424 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:32.466676 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:32.466928 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:32.468097 7379 hybrid_clock.cc:648] HybridClock initialized: now 1790313032468088 us; error 55 us; skew 500 ppm
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:32.470711 7379 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:32.471076 7379 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:32.471149 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:32.471303 7379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:32.472433 7379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "908be55eaad7004785d4a2ff64fefef9"
server_key_iv: "79eff0fe79d3b2729618fc09661c7162"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.472803 7379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "908be55eaad7004785d4a2ff64fefef9"
server_key_iv: "79eff0fe79d3b2729618fc09661c7162"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.474117 7379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:32.475005 7393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.475234 7379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:32.475349 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "908be55eaad7004785d4a2ff64fefef9"
server_key_iv: "79eff0fe79d3b2729618fc09661c7162"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.475427 7379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:32.494342 7379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:32.497013 7379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:32.497135 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:32.497335 7379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:32.497681 7379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:32.497718 7379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.497740 7379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:32.497790 7379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.504046 7379 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:36209
I20260925 05:10:32.504127 7506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:36209 every 8 connection(s)
I20260925 05:10:32.504410 7379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:32.513583 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.505153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:47385 (local address 127.3.6.190:44783)
0925 05:10:32.505228 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.505230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.505528 (+ 298us) server_negotiation.cc:407] Connection header received
0925 05:10:32.506017 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.506018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.506030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.506081 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:32.507015 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.507552 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.508608 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.508792 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.508844 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.511040 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:32.511042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:32.511042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:32.511049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:32.512379 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.512790 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.512790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.512791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.512840 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.513052 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.513052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.513052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.513151 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:32.513206 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.513430 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.513493 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:10:32.513727 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.505155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44783 (local address 127.3.6.130:47385)
0925 05:10:32.505375 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.505415 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.505926 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.506151 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.506154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.506422 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:32.506925 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.506930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.507636 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.507637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.508518 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.508520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.508865 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.508866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.508963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.509512 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:32.509521 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:32.510949 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:32.512455 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:32.512461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:32.512462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:32.512724 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:32.512900 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:32.512900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:32.512900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:32.512999 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:32.513205 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:32.513209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:32.513322 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.513405 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.513524 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260925 05:10:32.514235 7507 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44783
I20260925 05:10:32.514369 7507 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:32.514456 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 7379
I20260925 05:10:32.514554 7507 heartbeater.cc:511] Master 127.3.6.190:44783 requested a full tablet report, sending...
I20260925 05:10:32.514586 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:32.514793 3098 external_mini_cluster.cc:1428] Setting key baa1cf7480fd2a6daffe88d54ed4d4d3
I20260925 05:10:32.514974 7184 ts_manager.cc:194] Registered new tserver with Master: 1dc936ecb2fe4f4e9b1645cdb9087c6b (127.3.6.130:36209)
I20260925 05:10:32.515440 7184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:47385
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:32.543164 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44783
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:32.626293 7512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:32.626497 7512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:32.626520 7512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:32.627638 7512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:32.627684 7512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:32.627712 7512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:32.629226 7512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:32.629415 7512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:32.629590 7512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:32.634830 7517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.634812 7520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.634812 7518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:32.635522 7512 server_base.cc:1034] running on GCE node
I20260925 05:10:32.635823 7512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:32.636055 7512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:32.637229 7512 hybrid_clock.cc:648] HybridClock initialized: now 1790313032637221 us; error 38 us; skew 500 ppm
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:32.640244 7512 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:32.640640 7512 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:32.640707 7512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:32.640862 7512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:32.641995 7512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "070e527e528f486f960e810b9e42eed1"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "03ac57fc9bb649f0735dc480f07aecfe"
server_key_iv: "b6883e2cdedad2f614c649eebe25ef0c"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.642364 7512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "070e527e528f486f960e810b9e42eed1"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "03ac57fc9bb649f0735dc480f07aecfe"
server_key_iv: "b6883e2cdedad2f614c649eebe25ef0c"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.643877 7512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:32.644680 7526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.644851 7512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:32.644948 7512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "070e527e528f486f960e810b9e42eed1"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "03ac57fc9bb649f0735dc480f07aecfe"
server_key_iv: "b6883e2cdedad2f614c649eebe25ef0c"
server_key_version: "encryptionkey@0"
I20260925 05:10:32.645032 7512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:32.656526 7512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:32.658448 7512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:32.658617 7512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:32.658874 7512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:32.659191 7512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:32.659246 7512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.659288 7512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:32.659334 7512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:32.665741 7512 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:44983
I20260925 05:10:32.665783 7639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:44983 every 8 connection(s)
I20260925 05:10:32.666079 7512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:32.668591 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 7512
I20260925 05:10:32.668691 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:32.668915 3098 external_mini_cluster.cc:1428] Setting key 29867dd6b19c63da5977eeaada50c6d4
Sep 25 05:10:32 dist-test-slave-1lvv krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:32.675625 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.666865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:58453 (local address 127.3.6.190:44783)
0925 05:10:32.666912 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.666914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.667147 (+ 233us) server_negotiation.cc:407] Connection header received
0925 05:10:32.667655 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.667656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.667667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.667737 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:32.668926 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.669595 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.670669 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.670867 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.670921 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.673188 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:32.673190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:32.673190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:32.673197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:32.674412 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.674781 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.674782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.674782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.674833 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.675040 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.675040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.675040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.675142 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:32.675194 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.675493 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.675558 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:10:32.675918 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.666813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44783 (local address 127.3.6.131:58453)
0925 05:10:32.666988 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.667032 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.667564 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.667886 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.667890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.668191 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:32.668816 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.668823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.669691 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.669692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.670591 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.670594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.670942 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.670943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.671049 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.671636 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:32.671645 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:32.673102 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:32.674485 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:32.674491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:32.674493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:32.674715 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:32.674890 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:32.674890 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:32.674890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:32.674986 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:32.675233 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:32.675239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:32.675367 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.675470 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.675608 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260925 05:10:32.676520 7640 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44783
I20260925 05:10:32.676640 7640 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:32.676797 7640 heartbeater.cc:511] Master 127.3.6.190:44783 requested a full tablet report, sending...
I20260925 05:10:32.677155 7184 ts_manager.cc:194] Registered new tserver with Master: 070e527e528f486f960e810b9e42eed1 (127.3.6.131:44983)
I20260925 05:10:32.677539 7184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:58453
I20260925 05:10:32.680711 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:32.688099 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.682097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.3.6.190:44783)
0925 05:10:32.682176 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.682179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.682279 (+ 100us) server_negotiation.cc:407] Connection header received
0925 05:10:32.682351 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.682352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.682362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.682418 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:32.682985 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.683456 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.684289 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.684491 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.684575 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.685709 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:32.685711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:32.685712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:32.685720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:32.686866 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.687232 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.687233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.687233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.687296 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:32.687500 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:32.687501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:32.687501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:32.687656 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:32.687794 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.687916 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.687996 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:10:32.689580 7184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:32.691107 7184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:32.698160 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.693767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:44983 (local address 127.0.0.1:60028)
0925 05:10:32.694147 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.694160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.694187 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.694682 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.694683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.694685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:32.694727 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.694728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.696141 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.696142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.697012 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.697015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.697943 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.697943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.697999 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.698011 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.698066 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.698106 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":105,"threads_started":1}
I20260925 05:10:32.698210 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.694005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.3.6.131:44983)
0925 05:10:32.694523 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.694528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.694537 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:32.694573 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.694574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.694591 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.694670 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:32.694934 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.696035 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.697105 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.697858 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.697938 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.697998 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.698091 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.698130 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":89,"threads_started":1}
I20260925 05:10:32.698820 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.693766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:36209 (local address 127.0.0.1:54640)
0925 05:10:32.694026 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.694035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.694078 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.694390 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.694393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.694396 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:32.694467 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.694469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.695785 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.695786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.697089 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.697092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.698563 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.698564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.698627 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.698631 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.698707 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.698750 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":70,"thread_start_us":102,"threads_started":1}
I20260925 05:10:32.698843 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.694021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.3.6.130:36209)
0925 05:10:32.694187 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.694192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.694202 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:32.694272 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.694274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.694291 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.694378 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:32.694556 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.695668 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.697167 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.697858 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.697938 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.697998 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.698727 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.698766 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260925 05:10:32.699000 7574 tablet_service.cc:1467] Processing CreateTablet for tablet 69dc131264a649c78112a03f80b7ece8 (DEFAULT_TABLE table=test-table [id=5d8bea3799ff4b4f8d9459733d479ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:32.699154 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.693707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:42939 (local address 127.0.0.1:39620)
0925 05:10:32.693879 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.693924 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.693989 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.694557 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.694558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.694561 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:32.694624 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.694625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.695659 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.695660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.697092 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.697094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.698918 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.698920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.698983 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.698988 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.699051 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.699093 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":39,"thread_start_us":61,"threads_started":1}
I20260925 05:10:32.699422 7574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69dc131264a649c78112a03f80b7ece8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:32.699465 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.693833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.3.6.129:42939)
0925 05:10:32.694277 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.694281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.694311 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:10:32.694361 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.694363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.694379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.694459 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:32.694726 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.695546 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.698037 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.698754 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.698827 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.698880 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.699092 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.699139 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":81,"threads_started":1}
I20260925 05:10:32.699870 7441 tablet_service.cc:1467] Processing CreateTablet for tablet 69dc131264a649c78112a03f80b7ece8 (DEFAULT_TABLE table=test-table [id=5d8bea3799ff4b4f8d9459733d479ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:32.700219 7441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69dc131264a649c78112a03f80b7ece8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:32.700187 7308 tablet_service.cc:1467] Processing CreateTablet for tablet 69dc131264a649c78112a03f80b7ece8 (DEFAULT_TABLE table=test-table [id=5d8bea3799ff4b4f8d9459733d479ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:32.700467 7308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69dc131264a649c78112a03f80b7ece8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:32.703163 7657 tablet_bootstrap.cc:492] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Bootstrap starting.
I20260925 05:10:32.703517 7658 tablet_bootstrap.cc:492] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: Bootstrap starting.
I20260925 05:10:32.703655 7659 tablet_bootstrap.cc:492] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Bootstrap starting.
I20260925 05:10:32.704030 7657 tablet_bootstrap.cc:654] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:32.704231 7658 tablet_bootstrap.cc:654] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:32.704471 7659 tablet_bootstrap.cc:654] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:32.704697 7657 log.cc:824] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:32.705003 7658 log.cc:824] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:32.705104 7659 log.cc:824] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:32.705871 7658 tablet_bootstrap.cc:492] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: No bootstrap required, opened a new log
I20260925 05:10:32.705871 7659 tablet_bootstrap.cc:492] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: No bootstrap required, opened a new log
I20260925 05:10:32.705884 7657 tablet_bootstrap.cc:492] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: No bootstrap required, opened a new log
I20260925 05:10:32.705958 7657 ts_tablet_manager.cc:1397] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:10:32.705958 7659 ts_tablet_manager.cc:1397] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:32.705960 7658 ts_tablet_manager.cc:1397] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:10:32.707356 7658 raft_consensus.cc:348] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.707490 7658 raft_consensus.cc:374] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:32.707511 7658 raft_consensus.cc:729] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c765d3cda56742edaaed9a888b00a5ba, State: Initialized, Role: FOLLOWER
I20260925 05:10:32.707605 7658 consensus_queue.cc:260] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.707922 7374 heartbeater.cc:503] Master 127.3.6.190:44783 was elected leader, sending a full tablet report...
I20260925 05:10:32.707923 7658 ts_tablet_manager.cc:1428] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:32.708051 7659 raft_consensus.cc:348] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.708035 7657 raft_consensus.cc:348] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.708197 7659 raft_consensus.cc:374] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:32.708201 7657 raft_consensus.cc:374] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:32.708235 7657 raft_consensus.cc:729] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070e527e528f486f960e810b9e42eed1, State: Initialized, Role: FOLLOWER
I20260925 05:10:32.708232 7659 raft_consensus.cc:729] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc936ecb2fe4f4e9b1645cdb9087c6b, State: Initialized, Role: FOLLOWER
I20260925 05:10:32.708345 7657 consensus_queue.cc:260] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.708364 7659 consensus_queue.cc:260] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.708633 7640 heartbeater.cc:503] Master 127.3.6.190:44783 was elected leader, sending a full tablet report...
I20260925 05:10:32.708859 7659 ts_tablet_manager.cc:1428] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:32.709021 7507 heartbeater.cc:503] Master 127.3.6.190:44783 was elected leader, sending a full tablet report...
I20260925 05:10:32.709167 7657 ts_tablet_manager.cc:1428] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 05:10:32.755074 7508 tablet.cc:2369] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:32.810246 7663 raft_consensus.cc:482] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:32.810379 7663 raft_consensus.cc:504] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.810747 7663 leader_election.cc:288] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dc936ecb2fe4f4e9b1645cdb9087c6b (127.3.6.130:36209), 070e527e528f486f960e810b9e42eed1 (127.3.6.131:44983)
I20260925 05:10:32.813982 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.811063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:51891 (local address 127.3.6.130:36209)
0925 05:10:32.811133 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.811135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.811148 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:10:32.811200 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.811201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.811214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.811272 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:32.811436 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.812038 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.813236 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.813607 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.813696 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.813706 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.813874 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.813910 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:10:32.813982 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.810967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:36209 (local address 127.3.6.129:51891)
0925 05:10:32.811035 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.811048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.811095 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.811274 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.811275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.811277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:32.811356 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.811358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.812126 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.812127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.813151 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.813153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.813697 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.813698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.813781 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.813789 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.813865 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.813908 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:10:32.814065 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.811099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:52517 (local address 127.3.6.131:44983)
0925 05:10:32.811173 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.811175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.811191 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:10:32.811277 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.811278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.811290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.811353 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:32.811557 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.812125 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.813290 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.813671 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.813759 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.813769 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.813943 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.813981 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:10:32.814139 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.810999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:44983 (local address 127.3.6.129:52517)
0925 05:10:32.811132 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.811142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:32.811202 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:32.811369 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:32.811369 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:32.811371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:32.811468 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.811469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.812215 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.812216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.813206 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:32.813208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.813761 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.813761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:32.813866 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.813869 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:32.813936 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:10:32.813975 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260925 05:10:32.814497 7594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dc131264a649c78112a03f80b7ece8" candidate_uuid: "c765d3cda56742edaaed9a888b00a5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070e527e528f486f960e810b9e42eed1" is_pre_election: true
I20260925 05:10:32.814520 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dc131264a649c78112a03f80b7ece8" candidate_uuid: "c765d3cda56742edaaed9a888b00a5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" is_pre_election: true
I20260925 05:10:32.814674 7594 raft_consensus.cc:2454] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c765d3cda56742edaaed9a888b00a5ba in term 0.
I20260925 05:10:32.814674 7461 raft_consensus.cc:2454] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c765d3cda56742edaaed9a888b00a5ba in term 0.
I20260925 05:10:32.814913 7264 leader_election.cc:302] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070e527e528f486f960e810b9e42eed1, c765d3cda56742edaaed9a888b00a5ba; no voters:
I20260925 05:10:32.815066 7663 raft_consensus.cc:2781] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:32.815148 7663 raft_consensus.cc:482] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:32.815202 7663 raft_consensus.cc:3037] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:32.815927 7663 raft_consensus.cc:504] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.816085 7663 leader_election.cc:288] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [CANDIDATE]: Term 1 election: Requested vote from peers 1dc936ecb2fe4f4e9b1645cdb9087c6b (127.3.6.130:36209), 070e527e528f486f960e810b9e42eed1 (127.3.6.131:44983)
I20260925 05:10:32.816272 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dc131264a649c78112a03f80b7ece8" candidate_uuid: "c765d3cda56742edaaed9a888b00a5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b"
I20260925 05:10:32.816278 7594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69dc131264a649c78112a03f80b7ece8" candidate_uuid: "c765d3cda56742edaaed9a888b00a5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070e527e528f486f960e810b9e42eed1"
I20260925 05:10:32.816362 7461 raft_consensus.cc:3037] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:32.816362 7594 raft_consensus.cc:3037] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:32.817325 7594 raft_consensus.cc:2454] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c765d3cda56742edaaed9a888b00a5ba in term 1.
I20260925 05:10:32.817325 7461 raft_consensus.cc:2454] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c765d3cda56742edaaed9a888b00a5ba in term 1.
I20260925 05:10:32.817552 7262 leader_election.cc:302] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc936ecb2fe4f4e9b1645cdb9087c6b, c765d3cda56742edaaed9a888b00a5ba; no voters:
I20260925 05:10:32.817720 7663 raft_consensus.cc:2781] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:32.817909 7663 raft_consensus.cc:686] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 1 LEADER]: Becoming Leader. State: Replica: c765d3cda56742edaaed9a888b00a5ba, State: Running, Role: LEADER
I20260925 05:10:32.818027 7663 consensus_queue.cc:237] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } }
I20260925 05:10:32.819036 7182 catalog_manager.cc:5495] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba reported cstate change: term changed from 0 to 1, leader changed from <none> to c765d3cda56742edaaed9a888b00a5ba (127.3.6.129). New cstate: current_term: 1 leader_uuid: "c765d3cda56742edaaed9a888b00a5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c765d3cda56742edaaed9a888b00a5ba" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 } health_report { overall_health: UNKNOWN } } }
W20260925 05:10:32.830716 7375 tablet.cc:2369] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:32.846555 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.844034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.3.6.129:42939)
0925 05:10:32.844097 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:32.844099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:32.844109 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:32.844158 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:32.844160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:32.844187 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:32.844264 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:32.844438 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.844924 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.845716 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:32.845914 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:32.845993 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:32.846181 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:32.846285 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:32.846375 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:32.846417 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 05:10:32.846439 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:10:32.848666 7461 raft_consensus.cc:1261] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 1 FOLLOWER]: Refusing update from remote peer c765d3cda56742edaaed9a888b00a5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:32.848690 7594 raft_consensus.cc:1261] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 1 FOLLOWER]: Refusing update from remote peer c765d3cda56742edaaed9a888b00a5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:32.849112 7663 consensus_queue.cc:1035] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc936ecb2fe4f4e9b1645cdb9087c6b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:32.849290 7667 consensus_queue.cc:1035] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "070e527e528f486f960e810b9e42eed1" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:32.852869 7670 mvcc.cc:205] Tried to move back new op lower bound from 7333122182545149952 to 7333122182423302144. Current Snapshot: MvccSnapshot[applied={T|T < 7333122182545149952 or (T in {7333122182545149952})}]
I20260925 05:10:32.854342 7183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:32.854449 7183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:32.855355 7183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82d01b84634d4ddba071b8ba1aa4daca: Sending DeleteTablet for 3 replicas of tablet 69dc131264a649c78112a03f80b7ece8
I20260925 05:10:32.855589 7308 tablet_service.cc:1514] Processing DeleteTablet for tablet 69dc131264a649c78112a03f80b7ece8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:32 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:39620
I20260925 05:10:32.855751 7574 tablet_service.cc:1514] Processing DeleteTablet for tablet 69dc131264a649c78112a03f80b7ece8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:32 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:60028
I20260925 05:10:32.855796 7441 tablet_service.cc:1514] Processing DeleteTablet for tablet 69dc131264a649c78112a03f80b7ece8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:32 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:54640
I20260925 05:10:32.855614 7307 tablet_service.cc:1514] Processing DeleteTablet for tablet 69dc131264a649c78112a03f80b7ece8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:32 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:39620
I20260925 05:10:32.855894 7679 tablet_replica.cc:330] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba: stopping tablet replica
I20260925 05:10:32.856025 7679 raft_consensus.cc:2229] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:32.856108 7680 tablet_replica.cc:330] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: stopping tablet replica
I20260925 05:10:32.856220 7680 raft_consensus.cc:2229] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:32.856292 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 7246
I20260925 05:10:32.856326 7680 raft_consensus.cc:2258] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:32.856438 7679 raft_consensus.cc:2258] T 69dc131264a649c78112a03f80b7ece8 P c765d3cda56742edaaed9a888b00a5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:32.856468 7681 tablet_replica.cc:330] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: stopping tablet replica
I20260925 05:10:32.856556 7681 raft_consensus.cc:2229] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:32.856642 7681 raft_consensus.cc:2258] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:32.857204 7680 ts_tablet_manager.cc:1905] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:32.858534 7681 ts_tablet_manager.cc:1905] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:32.859236 7680 ts_tablet_manager.cc:1918] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:32.859313 7680 log.cc:1196] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/69dc131264a649c78112a03f80b7ece8
I20260925 05:10:32.859596 7680 ts_tablet_manager.cc:1939] T 69dc131264a649c78112a03f80b7ece8 P 1dc936ecb2fe4f4e9b1645cdb9087c6b: Deleting consensus metadata
I20260925 05:10:32.860009 7681 ts_tablet_manager.cc:1918] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:32.860105 7681 log.cc:1196] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/69dc131264a649c78112a03f80b7ece8
I20260925 05:10:32.860157 7169 catalog_manager.cc:4841] TS 1dc936ecb2fe4f4e9b1645cdb9087c6b (127.3.6.130:36209): tablet 69dc131264a649c78112a03f80b7ece8 (table test-table [id=5d8bea3799ff4b4f8d9459733d479ecb]) successfully deleted
I20260925 05:10:32.860419 7681 ts_tablet_manager.cc:1939] T 69dc131264a649c78112a03f80b7ece8 P 070e527e528f486f960e810b9e42eed1: Deleting consensus metadata
I20260925 05:10:32.860797 7171 catalog_manager.cc:4841] TS 070e527e528f486f960e810b9e42eed1 (127.3.6.131:44983): tablet 69dc131264a649c78112a03f80b7ece8 (table test-table [id=5d8bea3799ff4b4f8d9459733d479ecb]) successfully deleted
W20260925 05:10:32.862706 7170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:42939 (error 108)
I20260925 05:10:32.863263 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:32.863100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:42939 (local address 127.0.0.1:39632)
0925 05:10:32.863165 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:32.863223 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 05:10:32.863296 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 7379
W20260925 05:10:32.863405 7170 catalog_manager.cc:4584] TS c765d3cda56742edaaed9a888b00a5ba (127.3.6.129:42939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69dc131264a649c78112a03f80b7ece8: Network error: Client connection negotiation failed: client connection to 127.3.6.129:42939: connect: Connection refused (error 111)
I20260925 05:10:32.868741 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 7512
I20260925 05:10:32.873450 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 7153
2026-09-25T05:10:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5331 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T05:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:32Z Disabled control of system clock
I20260925 05:10:32.902740 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44435
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:10:32.986423 7686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:32.986676 7686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:32.986701 7686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:32.988049 7686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:32.988097 7686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:32.988112 7686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:32.988125 7686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:32.988138 7686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:32.989573 7686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:32.989861 7686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:32.990191 7686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:32.995298 7691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.995260 7692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:32.995306 7694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:32.995409 7686 server_base.cc:1034] running on GCE node
I20260925 05:10:32.995913 7686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:32.996250 7686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:32.997439 7686 hybrid_clock.cc:648] HybridClock initialized: now 1790313032997414 us; error 44 us; skew 500 ppm
I20260925 05:10:32.998879 7686 webserver.cc:460] Webserver started at http://127.3.6.190:44497/ using document root <none> and password file <none>
I20260925 05:10:32.999150 7686 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:32.999228 7686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:32.999385 7686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:33.000605 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "d82a7ea7b401409d9589e94e31e01974"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "0bb77fa10c9531acafde8d2c78403434"
server_key_iv: "b3b5f1caaeb452dffc02ef8c88865268"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.001003 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "d82a7ea7b401409d9589e94e31e01974"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "0bb77fa10c9531acafde8d2c78403434"
server_key_iv: "b3b5f1caaeb452dffc02ef8c88865268"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.002460 7686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:10:33.003239 7700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.003422 7686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:33.003535 7686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "d82a7ea7b401409d9589e94e31e01974"
format_stamp: "Formatted at 2026-09-25 05:10:32 on dist-test-slave-1lvv"
server_key: "0bb77fa10c9531acafde8d2c78403434"
server_key_iv: "b3b5f1caaeb452dffc02ef8c88865268"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.003628 7686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:33.007164 7686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:33.007486 7686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:33.007685 7686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:33.011690 7686 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44435
I20260925 05:10:33.011729 7752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44435 every 8 connection(s)
I20260925 05:10:33.012149 7686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:33.012871 7753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:33.015825 7753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974: Bootstrap starting.
I20260925 05:10:33.016462 7753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:33.016779 7753 log.cc:824] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:33.017635 7753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974: No bootstrap required, opened a new log
I20260925 05:10:33.017652 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 7686
I20260925 05:10:33.017772 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:33.017957 3098 external_mini_cluster.cc:1428] Setting key 219d558b26bf1b8685f4a706526a1e1e
I20260925 05:10:33.020038 7753 raft_consensus.cc:348] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82a7ea7b401409d9589e94e31e01974" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44435 } }
I20260925 05:10:33.020252 7753 raft_consensus.cc:374] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:33.020294 7753 raft_consensus.cc:729] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82a7ea7b401409d9589e94e31e01974, State: Initialized, Role: FOLLOWER
I20260925 05:10:33.020475 7753 consensus_queue.cc:260] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82a7ea7b401409d9589e94e31e01974" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44435 } }
I20260925 05:10:33.020905 7753 raft_consensus.cc:388] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:33.020956 7753 raft_consensus.cc:482] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:33.021006 7753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:33.021778 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.018584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.3.6.190:44435)
0925 05:10:33.018830 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.018840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.018875 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:10:33.019986 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.019997 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.020070 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.020237 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.020944 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.020948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.020948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.020953 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.021005 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.021022 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.021100 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.021207 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.021271 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260925 05:10:33.021878 7753 raft_consensus.cc:504] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82a7ea7b401409d9589e94e31e01974" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44435 } }
I20260925 05:10:33.021993 7753 leader_election.cc:302] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d82a7ea7b401409d9589e94e31e01974; no voters:
I20260925 05:10:33.022156 7753 leader_election.cc:288] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:33.022186 7758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:33.022372 7758 raft_consensus.cc:686] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [term 1 LEADER]: Becoming Leader. State: Replica: d82a7ea7b401409d9589e94e31e01974, State: Running, Role: LEADER
I20260925 05:10:33.022502 7753 sys_catalog.cc:564] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:33.022516 7758 consensus_queue.cc:237] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82a7ea7b401409d9589e94e31e01974" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44435 } }
I20260925 05:10:33.022936 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d82a7ea7b401409d9589e94e31e01974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82a7ea7b401409d9589e94e31e01974" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44435 } } }
I20260925 05:10:33.023017 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:33.023170 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d82a7ea7b401409d9589e94e31e01974. Latest consensus state: current_term: 1 leader_uuid: "d82a7ea7b401409d9589e94e31e01974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82a7ea7b401409d9589e94e31e01974" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44435 } } }
I20260925 05:10:33.023234 7762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:33.023516 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:33.024081 7762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:33.025666 7762 catalog_manager.cc:1347] Generated new cluster ID: 34240865e6044952804244e8f48101bd
I20260925 05:10:33.025720 7762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:33.035830 7762 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:33.036041 7762 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:33.039834 7762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974: Generated new TSK 0
I20260925 05:10:33.040109 7762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:10:33.042924 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44435
--builtin_ntp_servers=127.3.6.148:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:10:33.128366 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:33.128614 7777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:33.128654 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:33.129859 7777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:33.129923 7777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:33.129968 7777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:33.131349 7777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:33.131628 7777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:33.131999 7777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:33.138083 7782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:33.138319 7777 server_base.cc:1034] running on GCE node
W20260925 05:10:33.138099 7785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:33.138099 7783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:33.138646 7777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:33.138947 7777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:33.140108 7777 hybrid_clock.cc:648] HybridClock initialized: now 1790313033140095 us; error 29 us; skew 500 ppm
I20260925 05:10:33.141666 7777 webserver.cc:460] Webserver started at http://127.3.6.129:37561/ using document root <none> and password file <none>
I20260925 05:10:33.141939 7777 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:33.142004 7777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:33.142156 7777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:33.143613 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "87bb1bd7969f48a8b2dd837d68431662"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "45b7489cd72934353ae850a359eab5a8"
server_key_iv: "f7afa5194660a4d0a3d0661f51ec809a"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.144189 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "87bb1bd7969f48a8b2dd837d68431662"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "45b7489cd72934353ae850a359eab5a8"
server_key_iv: "f7afa5194660a4d0a3d0661f51ec809a"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.146076 7777 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:33.146942 7791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.147118 7777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:33.147238 7777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "87bb1bd7969f48a8b2dd837d68431662"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "45b7489cd72934353ae850a359eab5a8"
server_key_iv: "f7afa5194660a4d0a3d0661f51ec809a"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.147311 7777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:33.150856 7777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:33.151208 7777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:33.151377 7777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:33.151608 7777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:33.152006 7777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:33.152066 7777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.152112 7777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:33.152139 7777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.157788 7777 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:36209
I20260925 05:10:33.157876 7904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:36209 every 8 connection(s)
I20260925 05:10:33.158249 7777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:33.158715 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 7777
I20260925 05:10:33.158802 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:33.158944 3098 external_mini_cluster.cc:1428] Setting key 6f9d62b6fd031e1f10c27a8973c09f82
I20260925 05:10:33.160102 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44435
--builtin_ntp_servers=127.3.6.148:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 05:10:33.162199 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.159598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:43247 (local address 127.3.6.190:44435)
0925 05:10:33.159646 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.159648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.160011 (+ 363us) server_negotiation.cc:407] Connection header received
0925 05:10:33.160591 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.160592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.160601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.160683 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.161922 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.161923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.161924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.161930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.161951 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.161956 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.162035 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.162081 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.162100 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:33.162360 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.159627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44435 (local address 127.3.6.129:43247)
0925 05:10:33.159878 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.159915 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.160482 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.160742 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.160745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.160754 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.161763 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.161763 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.161794 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.161825 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.162029 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.162032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.162034 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.162087 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.162136 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":137,"threads_started":1}
I20260925 05:10:33.162815 7905 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44435
I20260925 05:10:33.162911 7905 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:33.163086 7905 heartbeater.cc:511] Master 127.3.6.190:44435 requested a full tablet report, sending...
I20260925 05:10:33.163609 7717 ts_manager.cc:194] Registered new tserver with Master: 87bb1bd7969f48a8b2dd837d68431662 (127.3.6.129:36209)
W20260925 05:10:33.243436 7908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:33.243686 7908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:33.243726 7908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:33.244940 7908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:33.245010 7908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:33.245056 7908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:33.246459 7908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:33.246730 7908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:33.247040 7908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:33.252367 7916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:33.252364 7914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:33.252398 7913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:33.252774 7908 server_base.cc:1034] running on GCE node
I20260925 05:10:33.252988 7908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:33.253248 7908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:33.254420 7908 hybrid_clock.cc:648] HybridClock initialized: now 1790313033254412 us; error 39 us; skew 500 ppm
I20260925 05:10:33.255906 7908 webserver.cc:460] Webserver started at http://127.3.6.130:37517/ using document root <none> and password file <none>
I20260925 05:10:33.256131 7908 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:33.256182 7908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:33.256275 7908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:33.257371 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "fe57cfc32a484b01973c99e7c835328b"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "c05949cfdb972ff61cf8bf158626acde"
server_key_iv: "45bdebe4e81078e415e5cbb68b7e2a42"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.257682 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "fe57cfc32a484b01973c99e7c835328b"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "c05949cfdb972ff61cf8bf158626acde"
server_key_iv: "45bdebe4e81078e415e5cbb68b7e2a42"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.259344 7908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:33.260180 7922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.260385 7908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:10:33.260492 7908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "fe57cfc32a484b01973c99e7c835328b"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "c05949cfdb972ff61cf8bf158626acde"
server_key_iv: "45bdebe4e81078e415e5cbb68b7e2a42"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.260579 7908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:33.264055 7908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:33.264335 7908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:33.264490 7908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:33.264708 7908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:33.265024 7908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:33.265080 7908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.265121 7908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:33.265148 7908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.271184 7908 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:34943
I20260925 05:10:33.271222 8035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:34943 every 8 connection(s)
I20260925 05:10:33.271574 7908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:33.274791 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.272364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:49655 (local address 127.3.6.190:44435)
0925 05:10:33.272439 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.272441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.272702 (+ 261us) server_negotiation.cc:407] Connection header received
0925 05:10:33.273238 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.273239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.273246 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.273306 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.274505 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.274507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.274508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.274513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.274531 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.274537 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.274615 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.274702 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.274718 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:10:33.275002 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.272358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44435 (local address 127.3.6.130:49655)
0925 05:10:33.272562 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.272619 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.273124 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.273383 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.273388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.273397 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.274332 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.274332 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.274362 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.274391 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.274612 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.274619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.274622 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.274712 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.274776 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260925 05:10:33.275449 8036 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44435
I20260925 05:10:33.275529 8036 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:33.275701 8036 heartbeater.cc:511] Master 127.3.6.190:44435 requested a full tablet report, sending...
I20260925 05:10:33.275893 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 7908
I20260925 05:10:33.276008 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:33.276211 3098 external_mini_cluster.cc:1428] Setting key ea7363e5f1bd05dc36d2953fac0c86f4
I20260925 05:10:33.276439 7717 ts_manager.cc:194] Registered new tserver with Master: fe57cfc32a484b01973c99e7c835328b (127.3.6.130:34943)
I20260925 05:10:33.277381 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44435
--builtin_ntp_servers=127.3.6.148:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:10:33.359828 8039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:33.360057 8039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:33.360080 8039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:33.361274 8039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:33.361315 8039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:33.361340 8039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:33.362738 8039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:33.363004 8039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:33.363319 8039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:33.368721 8045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:33.368829 8047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:33.369022 8044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:33.369040 8039 server_base.cc:1034] running on GCE node
I20260925 05:10:33.369287 8039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:33.369549 8039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:33.370719 8039 hybrid_clock.cc:648] HybridClock initialized: now 1790313033370708 us; error 46 us; skew 500 ppm
I20260925 05:10:33.372081 8039 webserver.cc:460] Webserver started at http://127.3.6.131:37025/ using document root <none> and password file <none>
I20260925 05:10:33.372351 8039 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:33.372424 8039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:33.372582 8039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:33.373664 8039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "6d29d402440b43568c89852202201e46"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "fe05135daad05f38dca09aaf1fe0452a"
server_key_iv: "8b2a4dd516d90d1bd66206814796c63c"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.374033 8039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "6d29d402440b43568c89852202201e46"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "fe05135daad05f38dca09aaf1fe0452a"
server_key_iv: "8b2a4dd516d90d1bd66206814796c63c"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.375375 8039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:33.376245 8053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.376451 8039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:10:33.376559 8039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "6d29d402440b43568c89852202201e46"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "fe05135daad05f38dca09aaf1fe0452a"
server_key_iv: "8b2a4dd516d90d1bd66206814796c63c"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.376633 8039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:33.380105 8039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:33.380412 8039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:33.380568 8039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:33.380779 8039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:33.381089 8039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:33.381142 8039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.381191 8039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:33.381220 8039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.387462 8039 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:34159
I20260925 05:10:33.387557 8166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:34159 every 8 connection(s)
I20260925 05:10:33.387984 8039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:33.390956 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.388793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:44147 (local address 127.3.6.190:44435)
0925 05:10:33.388883 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.388885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.389084 (+ 199us) server_negotiation.cc:407] Connection header received
0925 05:10:33.389554 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.389555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.389560 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.389617 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.390747 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.390749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.390749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.390752 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.390767 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.390771 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.390822 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.390888 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.390901 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:10:33.391207 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.388752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44435 (local address 127.3.6.131:44147)
0925 05:10:33.388955 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.388991 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.389462 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.389674 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.389678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.389687 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.390579 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.390580 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.390611 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.390640 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.390831 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.390835 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.390837 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.390895 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.390945 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260925 05:10:33.391678 8167 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44435
I20260925 05:10:33.391804 8167 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:33.391959 8167 heartbeater.cc:511] Master 127.3.6.190:44435 requested a full tablet report, sending...
I20260925 05:10:33.392309 7716 ts_manager.cc:194] Registered new tserver with Master: 6d29d402440b43568c89852202201e46 (127.3.6.131:34159)
I20260925 05:10:33.392584 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8039
I20260925 05:10:33.392697 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:33.392908 3098 external_mini_cluster.cc:1428] Setting key d42f397780fa7512f68ab08535ca6f00
I20260925 05:10:33.393739 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:10:33.396370 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.395131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.3.6.190:44435)
0925 05:10:33.395171 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.395173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.395325 (+ 152us) server_negotiation.cc:407] Connection header received
0925 05:10:33.395426 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.395427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.395433 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.395509 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.396140 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.396142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.396142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.396147 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.396164 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.396168 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.396215 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.396282 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.396295 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:33.398046 7716 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:33.400200 7716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:33.404762 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.402729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.3.6.130:34943)
0925 05:10:33.402930 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.402935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.403101 (+ 166us) server_negotiation.cc:407] Connection header received
0925 05:10:33.403292 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.403297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.403305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.403356 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.404425 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.404436 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.404436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.404441 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.404481 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.404509 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.404571 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.404646 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.404662 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260925 05:10:33.404791 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.402552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36209 (local address 127.0.0.1:42024)
0925 05:10:33.402990 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.403001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.403085 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.403644 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.403645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.403648 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.404337 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.404338 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.404342 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.404350 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.404613 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.404614 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.404614 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.404662 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.404676 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":113,"threads_started":1}
I20260925 05:10:33.404910 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.402724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.3.6.131:34159)
0925 05:10:33.402915 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.402921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.403061 (+ 140us) server_negotiation.cc:407] Connection header received
0925 05:10:33.403160 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.403163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.403174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.403250 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.404500 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.404508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.404508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.404512 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.404548 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.404620 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.404680 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.404802 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.404819 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260925 05:10:33.405043 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.402813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.3.6.129:36209)
0925 05:10:33.402987 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.402991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.403153 (+ 162us) server_negotiation.cc:407] Connection header received
0925 05:10:33.403183 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.403185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.403196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.403266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.404446 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.404456 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.404456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.404460 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.404496 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.404527 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.404585 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.404941 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.404958 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260925 05:10:33.404771 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.402547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:34943 (local address 127.0.0.1:48460)
0925 05:10:33.403039 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.403048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.403122 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.403665 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.403666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.403667 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.404283 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.404284 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.404317 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.404345 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.404563 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.404569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.404573 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.404639 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.404656 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":9,"thread_start_us":103,"threads_started":1}
I20260925 05:10:33.405224 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.402547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:34159 (local address 127.0.0.1:42878)
0925 05:10:33.402943 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.402985 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.403073 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.403616 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.403621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.403628 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.404403 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.404404 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.404413 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.404426 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.404706 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.404707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.404707 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.404751 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.404762 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":60,"threads_started":1}
I20260925 05:10:33.406090 7839 tablet_service.cc:1467] Processing CreateTablet for tablet 291bb1a5158e42edbd05605f12cbaae8 (DEFAULT_TABLE table=test-table [id=9ed6290003d7448285ec0fab50c6c6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:33.406090 7970 tablet_service.cc:1467] Processing CreateTablet for tablet 291bb1a5158e42edbd05605f12cbaae8 (DEFAULT_TABLE table=test-table [id=9ed6290003d7448285ec0fab50c6c6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:33.406512 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291bb1a5158e42edbd05605f12cbaae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:33.406512 7839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291bb1a5158e42edbd05605f12cbaae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:33.408514 8101 tablet_service.cc:1467] Processing CreateTablet for tablet 291bb1a5158e42edbd05605f12cbaae8 (DEFAULT_TABLE table=test-table [id=9ed6290003d7448285ec0fab50c6c6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:33.408881 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291bb1a5158e42edbd05605f12cbaae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:33.410446 8182 tablet_bootstrap.cc:492] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Bootstrap starting.
I20260925 05:10:33.411164 8182 tablet_bootstrap.cc:654] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:33.411512 8182 log.cc:824] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:33.411859 8181 tablet_bootstrap.cc:492] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: Bootstrap starting.
I20260925 05:10:33.412015 8184 tablet_bootstrap.cc:492] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Bootstrap starting.
I20260925 05:10:33.412637 8181 tablet_bootstrap.cc:654] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:33.412662 8184 tablet_bootstrap.cc:654] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:33.412751 8182 tablet_bootstrap.cc:492] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: No bootstrap required, opened a new log
I20260925 05:10:33.412845 8182 ts_tablet_manager.cc:1397] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:10:33.412969 8181 log.cc:824] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:33.413015 8184 log.cc:824] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:33.413781 8181 tablet_bootstrap.cc:492] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: No bootstrap required, opened a new log
I20260925 05:10:33.413861 8181 ts_tablet_manager.cc:1397] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 05:10:33.413928 8184 tablet_bootstrap.cc:492] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: No bootstrap required, opened a new log
I20260925 05:10:33.414106 8184 ts_tablet_manager.cc:1397] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:10:33.415233 8182 raft_consensus.cc:348] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.415374 8182 raft_consensus.cc:374] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:33.415397 8182 raft_consensus.cc:729] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe57cfc32a484b01973c99e7c835328b, State: Initialized, Role: FOLLOWER
I20260925 05:10:33.415526 8182 consensus_queue.cc:260] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.415846 8036 heartbeater.cc:503] Master 127.3.6.190:44435 was elected leader, sending a full tablet report...
I20260925 05:10:33.415846 8182 ts_tablet_manager.cc:1428] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:10:33.416584 8181 raft_consensus.cc:348] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.416811 8181 raft_consensus.cc:374] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:33.416857 8181 raft_consensus.cc:729] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bb1bd7969f48a8b2dd837d68431662, State: Initialized, Role: FOLLOWER
I20260925 05:10:33.417002 8181 consensus_queue.cc:260] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.417301 8181 ts_tablet_manager.cc:1428] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 05:10:33.417374 7905 heartbeater.cc:503] Master 127.3.6.190:44435 was elected leader, sending a full tablet report...
I20260925 05:10:33.417312 8184 raft_consensus.cc:348] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.417459 8184 raft_consensus.cc:374] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:33.417498 8184 raft_consensus.cc:729] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d29d402440b43568c89852202201e46, State: Initialized, Role: FOLLOWER
I20260925 05:10:33.417619 8184 consensus_queue.cc:260] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.417906 8167 heartbeater.cc:503] Master 127.3.6.190:44435 was elected leader, sending a full tablet report...
I20260925 05:10:33.417912 8184 ts_tablet_manager.cc:1428] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260925 05:10:33.522287 8037 tablet.cc:2369] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:10:33.638685 8168 tablet.cc:2369] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:10:33.658962 7906 tablet.cc:2369] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:33.724484 8187 raft_consensus.cc:482] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:33.724622 8187 raft_consensus.cc:504] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.725057 8187 leader_election.cc:288] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87bb1bd7969f48a8b2dd837d68431662 (127.3.6.129:36209), 6d29d402440b43568c89852202201e46 (127.3.6.131:34159)
I20260925 05:10:33.726519 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.725298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:34159 (local address 127.3.6.130:39571)
0925 05:10:33.725360 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.725374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.725425 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.725609 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.725610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.725613 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.726110 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.726111 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.726120 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.726134 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.726351 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.726352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.726353 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.726421 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.726436 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 05:10:33.726550 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.725435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:39571 (local address 127.3.6.131:34159)
0925 05:10:33.725483 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.725485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.725494 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:33.725523 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.725524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.725535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.725602 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.726237 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.726239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.726239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.726258 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.726277 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.726283 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.726356 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.726437 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.726449 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:10:33.726704 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.725328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36209 (local address 127.3.6.130:37665)
0925 05:10:33.725516 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.725526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.725633 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.725783 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.725784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.725786 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:33.726307 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:10:33.726308 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:33.726315 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:10:33.726328 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:33.726511 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:33.726512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:33.726512 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 05:10:33.726592 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:10:33.726607 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":103,"threads_started":1}
I20260925 05:10:33.726773 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.725439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:37665 (local address 127.3.6.129:36209)
0925 05:10:33.725488 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.725490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.725627 (+ 137us) server_negotiation.cc:407] Connection header received
0925 05:10:33.725695 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.725696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.725701 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.725779 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.726416 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.726418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.726418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.726423 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.726440 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.726445 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.726502 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.726577 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.726707 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:33.726866 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291bb1a5158e42edbd05605f12cbaae8" candidate_uuid: "fe57cfc32a484b01973c99e7c835328b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d29d402440b43568c89852202201e46" is_pre_election: true
I20260925 05:10:33.727025 8121 raft_consensus.cc:2454] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe57cfc32a484b01973c99e7c835328b in term 0.
I20260925 05:10:33.727151 7859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291bb1a5158e42edbd05605f12cbaae8" candidate_uuid: "fe57cfc32a484b01973c99e7c835328b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87bb1bd7969f48a8b2dd837d68431662" is_pre_election: true
I20260925 05:10:33.727339 7859 raft_consensus.cc:2454] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe57cfc32a484b01973c99e7c835328b in term 0.
I20260925 05:10:33.727353 7924 leader_election.cc:302] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d29d402440b43568c89852202201e46, fe57cfc32a484b01973c99e7c835328b; no voters:
I20260925 05:10:33.727475 8187 raft_consensus.cc:2781] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:33.727535 8187 raft_consensus.cc:482] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:33.727557 8187 raft_consensus.cc:3037] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:33.728370 8187 raft_consensus.cc:504] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.728515 8187 leader_election.cc:288] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [CANDIDATE]: Term 1 election: Requested vote from peers 87bb1bd7969f48a8b2dd837d68431662 (127.3.6.129:36209), 6d29d402440b43568c89852202201e46 (127.3.6.131:34159)
I20260925 05:10:33.728654 7859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291bb1a5158e42edbd05605f12cbaae8" candidate_uuid: "fe57cfc32a484b01973c99e7c835328b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87bb1bd7969f48a8b2dd837d68431662"
I20260925 05:10:33.728715 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291bb1a5158e42edbd05605f12cbaae8" candidate_uuid: "fe57cfc32a484b01973c99e7c835328b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d29d402440b43568c89852202201e46"
I20260925 05:10:33.728754 7859 raft_consensus.cc:3037] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:33.728824 8121 raft_consensus.cc:3037] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:33.729485 7859 raft_consensus.cc:2454] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe57cfc32a484b01973c99e7c835328b in term 1.
I20260925 05:10:33.729617 8121 raft_consensus.cc:2454] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe57cfc32a484b01973c99e7c835328b in term 1.
I20260925 05:10:33.729732 7926 leader_election.cc:302] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87bb1bd7969f48a8b2dd837d68431662, fe57cfc32a484b01973c99e7c835328b; no voters:
I20260925 05:10:33.729852 8187 raft_consensus.cc:2781] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:33.729995 8187 raft_consensus.cc:686] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 1 LEADER]: Becoming Leader. State: Replica: fe57cfc32a484b01973c99e7c835328b, State: Running, Role: LEADER
I20260925 05:10:33.730113 8187 consensus_queue.cc:237] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } }
I20260925 05:10:33.730976 7717 catalog_manager.cc:5495] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b reported cstate change: term changed from 0 to 1, leader changed from <none> to fe57cfc32a484b01973c99e7c835328b (127.3.6.130). New cstate: current_term: 1 leader_uuid: "fe57cfc32a484b01973c99e7c835328b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe57cfc32a484b01973c99e7c835328b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34943 } health_report { overall_health: HEALTHY } } }
I20260925 05:10:33.758005 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.757035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.3.6.130:34943)
0925 05:10:33.757106 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.757108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.757120 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:33.757159 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.757160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.757187 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.757241 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.757795 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.757796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.757797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.757801 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.757818 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.757822 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.757870 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.757937 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.757949 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:10:33.760169 7859 raft_consensus.cc:1261] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 1 FOLLOWER]: Refusing update from remote peer fe57cfc32a484b01973c99e7c835328b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:33.760175 8121 raft_consensus.cc:1261] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 1 FOLLOWER]: Refusing update from remote peer fe57cfc32a484b01973c99e7c835328b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:33.760560 8191 consensus_queue.cc:1035] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d29d402440b43568c89852202201e46" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:33.760687 8187 consensus_queue.cc:1035] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [LEADER]: Connected to new peer: Peer: permanent_uuid: "87bb1bd7969f48a8b2dd837d68431662" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:33.763250 8196 mvcc.cc:205] Tried to move back new op lower bound from 7333122186278432768 to 7333122186159185920. Current Snapshot: MvccSnapshot[applied={T|T < 7333122186278432768}]
I20260925 05:10:33.763830 8195 mvcc.cc:205] Tried to move back new op lower bound from 7333122186278432768 to 7333122186159185920. Current Snapshot: MvccSnapshot[applied={T|T < 7333122186278432768}]
I20260925 05:10:33.765460 7716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:33.765569 7716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:33.766330 7716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d82a7ea7b401409d9589e94e31e01974: Sending DeleteTablet for 3 replicas of tablet 291bb1a5158e42edbd05605f12cbaae8
I20260925 05:10:33.766659 7970 tablet_service.cc:1514] Processing DeleteTablet for tablet 291bb1a5158e42edbd05605f12cbaae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:33 UTC) from {username='slave'} at 127.0.0.1:48460
I20260925 05:10:33.766680 7839 tablet_service.cc:1514] Processing DeleteTablet for tablet 291bb1a5158e42edbd05605f12cbaae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:33 UTC) from {username='slave'} at 127.0.0.1:42024
I20260925 05:10:33.766906 8205 tablet_replica.cc:330] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662: stopping tablet replica
I20260925 05:10:33.766912 8101 tablet_service.cc:1514] Processing DeleteTablet for tablet 291bb1a5158e42edbd05605f12cbaae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:33 UTC) from {username='slave'} at 127.0.0.1:42878
I20260925 05:10:33.767036 8205 raft_consensus.cc:2229] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:33.767164 8205 raft_consensus.cc:2258] T 291bb1a5158e42edbd05605f12cbaae8 P 87bb1bd7969f48a8b2dd837d68431662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:33.767167 8206 tablet_replica.cc:330] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: stopping tablet replica
I20260925 05:10:33.767248 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 7777
I20260925 05:10:33.767366 8204 tablet_replica.cc:330] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: stopping tablet replica
I20260925 05:10:33.767450 8204 raft_consensus.cc:2229] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:33.767529 8204 raft_consensus.cc:2258] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:33.767782 8206 raft_consensus.cc:2229] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:33.767891 8206 raft_consensus.cc:2258] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:33.768595 8206 ts_tablet_manager.cc:1905] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:33.768970 8204 ts_tablet_manager.cc:1905] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:33.770609 8206 ts_tablet_manager.cc:1918] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:33.770700 8206 log.cc:1196] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/291bb1a5158e42edbd05605f12cbaae8
I20260925 05:10:33.770965 8206 ts_tablet_manager.cc:1939] T 291bb1a5158e42edbd05605f12cbaae8 P 6d29d402440b43568c89852202201e46: Deleting consensus metadata
I20260925 05:10:33.771013 8204 ts_tablet_manager.cc:1918] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:33.771132 8204 log.cc:1196] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/291bb1a5158e42edbd05605f12cbaae8
I20260925 05:10:33.771396 8204 ts_tablet_manager.cc:1939] T 291bb1a5158e42edbd05605f12cbaae8 P fe57cfc32a484b01973c99e7c835328b: Deleting consensus metadata
I20260925 05:10:33.771435 7702 catalog_manager.cc:4841] TS 6d29d402440b43568c89852202201e46 (127.3.6.131:34159): tablet 291bb1a5158e42edbd05605f12cbaae8 (table test-table [id=9ed6290003d7448285ec0fab50c6c6b8]) successfully deleted
I20260925 05:10:33.771883 7703 catalog_manager.cc:4841] TS fe57cfc32a484b01973c99e7c835328b (127.3.6.130:34943): tablet 291bb1a5158e42edbd05605f12cbaae8 (table test-table [id=9ed6290003d7448285ec0fab50c6c6b8]) successfully deleted
W20260925 05:10:33.773940 7704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:36209 (error 108)
I20260925 05:10:33.774488 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.774340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36209 (local address 127.0.0.1:45110)
0925 05:10:33.774405 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.774452 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 05:10:33.774564 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 7908
W20260925 05:10:33.776113 7704 catalog_manager.cc:4584] TS 87bb1bd7969f48a8b2dd837d68431662 (127.3.6.129:36209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 291bb1a5158e42edbd05605f12cbaae8: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36209: connect: Connection refused (error 111)
I20260925 05:10:33.779984 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8039
I20260925 05:10:33.785136 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 7686
2026-09-25T05:10:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (911 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 05:10:33.805891 3098 mini_oidc.cc:152] Starting JWKS server
I20260925 05:10:33.806982 3098 webserver.cc:460] Webserver started at https://127.0.0.1:41839/ using document root <none> and password file <none>
I20260925 05:10:33.807066 3098 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:10:33.807220 3098 webserver.cc:460] Webserver started at http://127.0.0.1:37105/ using document root <none> and password file <none>
2026-09-25T05:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:33Z Disabled control of system clock
I20260925 05:10:33.815997 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:41605
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:41605
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41839/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/testchainca.cert with env {}
W20260925 05:10:33.899257 8213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:33.899542 8213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:33.899593 8213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:33.900866 8213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:33.900956 8213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:33.900978 8213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:33.901000 8213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:33.901032 8213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:10:33.901055 8213 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:10:33.901084 8213 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41839/jwks.json
I20260925 05:10:33.902806 8213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:41605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41839/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:33.903117 8213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:33.903350 8213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:33.908499 8221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:33.910269 8218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:33.910689 8213 server_base.cc:1034] running on GCE node
W20260925 05:10:33.910831 8219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:33.911144 8213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:33.911461 8213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:33.912606 8213 hybrid_clock.cc:648] HybridClock initialized: now 1790313033912587 us; error 38 us; skew 500 ppm
I20260925 05:10:33.913949 8213 webserver.cc:460] Webserver started at http://127.3.6.190:33633/ using document root <none> and password file <none>
I20260925 05:10:33.914232 8213 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:33.914283 8213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:33.914436 8213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:33.915597 8213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "774a212fdf714f0ab1922d4faa685f6a"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "2f71001351d36c564a76524c2f8fdbc6"
server_key_iv: "64afbad2f2f641fdf7530c1086a37151"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.916044 8213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "774a212fdf714f0ab1922d4faa685f6a"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "2f71001351d36c564a76524c2f8fdbc6"
server_key_iv: "64afbad2f2f641fdf7530c1086a37151"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.917404 8213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:33.918218 8227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:33.918416 8213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:33.918514 8213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "774a212fdf714f0ab1922d4faa685f6a"
format_stamp: "Formatted at 2026-09-25 05:10:33 on dist-test-slave-1lvv"
server_key: "2f71001351d36c564a76524c2f8fdbc6"
server_key_iv: "64afbad2f2f641fdf7530c1086a37151"
server_key_version: "encryptionkey@0"
I20260925 05:10:33.918602 8213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:33.929345 8213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:33.929742 8213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:33.929916 8213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:33.934123 8213 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:41605
I20260925 05:10:33.934161 8279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:41605 every 8 connection(s)
I20260925 05:10:33.934530 8213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:33.935547 8280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:33.937927 8280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a: Bootstrap starting.
I20260925 05:10:33.938501 8280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:33.938750 8280 log.cc:824] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:33.939400 8280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a: No bootstrap required, opened a new log
I20260925 05:10:33.940673 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8213
I20260925 05:10:33.940799 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:33.940982 3098 external_mini_cluster.cc:1428] Setting key 055b2a397bf9467c605c786605a5f1ec
I20260925 05:10:33.941185 8280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774a212fdf714f0ab1922d4faa685f6a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41605 } }
I20260925 05:10:33.941365 8280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:33.941406 8280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774a212fdf714f0ab1922d4faa685f6a, State: Initialized, Role: FOLLOWER
I20260925 05:10:33.941578 8280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774a212fdf714f0ab1922d4faa685f6a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41605 } }
I20260925 05:10:33.941679 8280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:33.941711 8280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:33.941812 8280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:33.942570 8280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774a212fdf714f0ab1922d4faa685f6a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41605 } }
I20260925 05:10:33.942691 8280 leader_election.cc:302] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 774a212fdf714f0ab1922d4faa685f6a; no voters:
I20260925 05:10:33.942873 8280 leader_election.cc:288] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:33.942945 8285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:33.943112 8285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [term 1 LEADER]: Becoming Leader. State: Replica: 774a212fdf714f0ab1922d4faa685f6a, State: Running, Role: LEADER
I20260925 05:10:33.943132 8280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:33.943235 8285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774a212fdf714f0ab1922d4faa685f6a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41605 } }
I20260925 05:10:33.943553 8285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774a212fdf714f0ab1922d4faa685f6a. Latest consensus state: current_term: 1 leader_uuid: "774a212fdf714f0ab1922d4faa685f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774a212fdf714f0ab1922d4faa685f6a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41605 } } }
I20260925 05:10:33.943627 8285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:33.943809 8286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "774a212fdf714f0ab1922d4faa685f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774a212fdf714f0ab1922d4faa685f6a" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41605 } } }
I20260925 05:10:33.943879 8286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:33.943995 8293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:33.944707 8293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:33.946766 8293 catalog_manager.cc:1347] Generated new cluster ID: 424fea1746784a41a3086f3da20c30a4
I20260925 05:10:33.946837 8293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:33.947657 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.941469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.3.6.190:41605)
0925 05:10:33.941713 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.941723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.941806 (+ 83us) server_negotiation.cc:407] Connection header received
0925 05:10:33.941932 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.941939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.942012 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.942182 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:33.942386 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:33.944325 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:33.945464 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:33.946113 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:33.946239 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:33.946911 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:33.946915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:33.946915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:33.947127 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:33.947168 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:33.947184 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:33.947271 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:33.947359 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 05:10:33.947437 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260925 05:10:33.948583 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:10:33.961545 8293 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:33.962064 8293 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:33.970494 8293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 774a212fdf714f0ab1922d4faa685f6a: Generated new TSK 0
I20260925 05:10:33.970719 8293 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 05:10:34.002094 8310 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 05:10:34.002254 8310 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:10:34.002288 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:33.978835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.3.6.190:41605)
0925 05:10:33.978883 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:33.978885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:33.979021 (+ 136us) server_negotiation.cc:407] Connection header received
0925 05:10:33.979128 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:33.979129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:33.979139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:33.979202 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:10:33.979363 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:33.979904 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:33.981054 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:33.981256 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:33.981314 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:33.981702 (+ 388us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:10:34.002007 (+ 20305us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:10:34.002213 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 05:10:34.002350 8284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 05:10:34.002688 8309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:33.978718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:41605 (local address 127.0.0.1:35828)
0925 05:10:33.978938 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:33.978951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:33.979028 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:33.979222 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:33.979223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:33.979224 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:10:33.979296 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:33.979299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:33.979979 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:33.979980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:33.980952 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:33.980956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:33.981337 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:33.981341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:33.981412 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:33.981439 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:10:34.002191 (+ 20752us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:10:34.002335 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.6.190:41605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.6.190:41605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 05:10:34.003135 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8213
2026-09-25T05:10:34Z chronyd exiting
I20260925 05:10:34.039608 3098 test_util.cc:175] -----------------------------------------------
I20260925 05:10:34.039733 3098 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790313008371521-3098-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 05:10:34.040572 3098 mini_oidc.cc:152] Starting JWKS server
I20260925 05:10:34.041369 3098 webserver.cc:460] Webserver started at https://127.0.0.1:32791/ using document root <none> and password file <none>
I20260925 05:10:34.041445 3098 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:10:34.041637 3098 webserver.cc:460] Webserver started at http://127.0.0.1:36701/ using document root <none> and password file <none>
2026-09-25T05:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:34Z Disabled control of system clock
I20260925 05:10:34.050062 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44553
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44553
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32791/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/testchainca.cert with env {}
W20260925 05:10:34.131376 8316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:34.131650 8316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:34.131692 8316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:34.132918 8316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:34.132989 8316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:34.133011 8316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:34.133033 8316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:34.133054 8316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:10:34.133081 8316 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:10:34.133106 8316 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32791/jwks.json
I20260925 05:10:34.134691 8316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32791/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:34.135010 8316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:34.135257 8316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:34.140216 8324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:34.140216 8321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:34.140338 8316 server_base.cc:1034] running on GCE node
W20260925 05:10:34.140203 8322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:34.140761 8316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:34.141093 8316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:34.142278 8316 hybrid_clock.cc:648] HybridClock initialized: now 1790313034142263 us; error 41 us; skew 500 ppm
I20260925 05:10:34.143571 8316 webserver.cc:460] Webserver started at http://127.3.6.190:33185/ using document root <none> and password file <none>
I20260925 05:10:34.143882 8316 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:34.143957 8316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:34.144212 8316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:34.145344 8316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "19afbff96c0247e58f8b604f089bfff1"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "719146ad1ffc8efb796f995f706bc976"
server_key_iv: "c2044ab62a0bda1703651be25c791f5c"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.145745 8316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "19afbff96c0247e58f8b604f089bfff1"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "719146ad1ffc8efb796f995f706bc976"
server_key_iv: "c2044ab62a0bda1703651be25c791f5c"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.147121 8316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:10:34.147936 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:34.148123 8316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:34.148226 8316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "19afbff96c0247e58f8b604f089bfff1"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "719146ad1ffc8efb796f995f706bc976"
server_key_iv: "c2044ab62a0bda1703651be25c791f5c"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.148319 8316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:34.171208 8316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:34.171594 8316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:34.171794 8316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:34.176069 8316 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44553
I20260925 05:10:34.176105 8382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44553 every 8 connection(s)
I20260925 05:10:34.176473 8316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:34.177451 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:34.179937 8383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1: Bootstrap starting.
I20260925 05:10:34.180548 8383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:34.180796 8383 log.cc:824] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:34.181550 8383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1: No bootstrap required, opened a new log
I20260925 05:10:34.183182 8383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19afbff96c0247e58f8b604f089bfff1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44553 } }
I20260925 05:10:34.183331 8383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:34.183354 8383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19afbff96c0247e58f8b604f089bfff1, State: Initialized, Role: FOLLOWER
I20260925 05:10:34.183518 8383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19afbff96c0247e58f8b604f089bfff1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44553 } }
I20260925 05:10:34.183607 8383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:34.183630 8383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:34.183662 8383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:34.184266 8383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19afbff96c0247e58f8b604f089bfff1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44553 } }
I20260925 05:10:34.184360 8383 leader_election.cc:302] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19afbff96c0247e58f8b604f089bfff1; no voters:
I20260925 05:10:34.184504 8383 leader_election.cc:288] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:34.184583 8386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:34.184777 8386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [term 1 LEADER]: Becoming Leader. State: Replica: 19afbff96c0247e58f8b604f089bfff1, State: Running, Role: LEADER
I20260925 05:10:34.184824 8383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:34.184904 8386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19afbff96c0247e58f8b604f089bfff1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44553 } }
I20260925 05:10:34.185293 8387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19afbff96c0247e58f8b604f089bfff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19afbff96c0247e58f8b604f089bfff1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44553 } } }
I20260925 05:10:34.185391 8387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:34.185407 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19afbff96c0247e58f8b604f089bfff1. Latest consensus state: current_term: 1 leader_uuid: "19afbff96c0247e58f8b604f089bfff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19afbff96c0247e58f8b604f089bfff1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44553 } } }
I20260925 05:10:34.185413 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8316
I20260925 05:10:34.185457 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:34.185530 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:34.185707 8394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:34.185741 3098 external_mini_cluster.cc:1428] Setting key 5bbb6c8735d6a4d15345b3755a41e35c
I20260925 05:10:34.186281 8394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:34.188030 8394 catalog_manager.cc:1347] Generated new cluster ID: 826e1c2181094deda874ee96f4392622
I20260925 05:10:34.188090 8394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:34.191601 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.186346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.3.6.190:44553)
0925 05:10:34.186686 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:34.186696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:34.186819 (+ 123us) server_negotiation.cc:407] Connection header received
0925 05:10:34.187044 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:34.187052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:34.187124 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:34.187280 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:34.187594 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.188702 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.189551 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.189997 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.190148 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.190860 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:34.190864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:34.190865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:34.191067 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:34.191099 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:34.191110 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:34.191191 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:34.191267 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 05:10:34.191443 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":77,"threads_started":1}
I20260925 05:10:34.192484 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:10:34.207056 8394 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:34.207625 8394 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:34.210240 8394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19afbff96c0247e58f8b604f089bfff1: Generated new TSK 0
I20260925 05:10:34.210445 8394 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 05:10:34.231208 8413 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 05:10:34.231307 8413 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:10:34.231669 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.216791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.3.6.190:44553)
0925 05:10:34.216838 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:34.216841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:34.216933 (+ 92us) server_negotiation.cc:407] Connection header received
0925 05:10:34.217005 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:34.217006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:34.217020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:34.217092 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:10:34.217223 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.218051 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.218879 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.219203 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.219280 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.219490 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:10:34.231424 (+ 11934us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:10:34.231595 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 05:10:34.231737 8405 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 05:10:34.231755 8412 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:34.216682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44553 (local address 127.0.0.1:38102)
0925 05:10:34.216850 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:34.216861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:34.216929 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:34.217099 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:34.217100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:34.217101 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:10:34.217156 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:34.217158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.218170 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.218172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:34.218788 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:34.218790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.219307 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.219311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:34.219366 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.219372 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:10:34.231564 (+ 12192us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:10:34.231655 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.6.190:44553: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":117,"threads_started":1}
I20260925 05:10:34.232254 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8316
2026-09-25T05:10:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 05:10:34.270311 3098 mini_oidc.cc:152] Starting JWKS server
I20260925 05:10:34.271037 3098 webserver.cc:460] Webserver started at https://127.0.0.1:37543/ using document root <none> and password file <none>
I20260925 05:10:34.271111 3098 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:10:34.271288 3098 webserver.cc:460] Webserver started at http://127.0.0.1:35247/ using document root <none> and password file <none>
2026-09-25T05:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:34Z Disabled control of system clock
I20260925 05:10:34.280085 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:35451
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:35451
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37543/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:10:34.360728 8419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:34.360972 8419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:34.360996 8419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:34.362210 8419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:34.362258 8419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:34.362272 8419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:34.362283 8419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:34.362339 8419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:10:34.362356 8419 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:10:34.362382 8419 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37543/jwks.json
I20260925 05:10:34.363808 8419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:35451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37543/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:34.364077 8419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:34.364382 8419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:34.369546 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:34.369566 8425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:34.369546 8427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:34.369822 8419 server_base.cc:1034] running on GCE node
I20260925 05:10:34.370074 8419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:34.370378 8419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:34.371533 8419 hybrid_clock.cc:648] HybridClock initialized: now 1790313034371511 us; error 38 us; skew 500 ppm
I20260925 05:10:34.372951 8419 webserver.cc:460] Webserver started at http://127.3.6.190:41301/ using document root <none> and password file <none>
I20260925 05:10:34.373214 8419 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:34.373266 8419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:34.373443 8419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:34.374583 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "941309d28e004b6aa8b4b81cbc4e46d4"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "108792c25d9e247ca8a4288e0bb41967"
server_key_iv: "65c1c11310b00a3c58e327bfcaa844fc"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.374965 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "941309d28e004b6aa8b4b81cbc4e46d4"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "108792c25d9e247ca8a4288e0bb41967"
server_key_iv: "65c1c11310b00a3c58e327bfcaa844fc"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.376417 8419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:34.377148 8433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:34.377327 8419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:34.377435 8419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "941309d28e004b6aa8b4b81cbc4e46d4"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "108792c25d9e247ca8a4288e0bb41967"
server_key_iv: "65c1c11310b00a3c58e327bfcaa844fc"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.377511 8419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:34.401584 8419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:34.402029 8419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:34.402215 8419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:34.407195 8419 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:35451
I20260925 05:10:34.407236 8485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:35451 every 8 connection(s)
I20260925 05:10:34.407613 8419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:34.408723 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:34.411182 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4: Bootstrap starting.
I20260925 05:10:34.411785 8486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:34.412065 8486 log.cc:824] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:34.412771 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4: No bootstrap required, opened a new log
I20260925 05:10:34.414376 8486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35451 } }
I20260925 05:10:34.414535 8486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:34.414598 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8419
I20260925 05:10:34.414562 8486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941309d28e004b6aa8b4b81cbc4e46d4, State: Initialized, Role: FOLLOWER
I20260925 05:10:34.414747 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:34.414858 8486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35451 } }
I20260925 05:10:34.414924 3098 external_mini_cluster.cc:1428] Setting key 3aadb8e877b40e56828e02a4219e334d
I20260925 05:10:34.414934 8486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:34.415002 8486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:34.415063 8486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:34.415961 8486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35451 } }
I20260925 05:10:34.416092 8486 leader_election.cc:302] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 941309d28e004b6aa8b4b81cbc4e46d4; no voters:
I20260925 05:10:34.416270 8486 leader_election.cc:288] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:34.416304 8491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:34.416421 8491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [term 1 LEADER]: Becoming Leader. State: Replica: 941309d28e004b6aa8b4b81cbc4e46d4, State: Running, Role: LEADER
I20260925 05:10:34.416512 8491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35451 } }
I20260925 05:10:34.416574 8486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:34.416886 8491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35451 } } }
I20260925 05:10:34.416986 8491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:34.417138 8491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 941309d28e004b6aa8b4b81cbc4e46d4. Latest consensus state: current_term: 1 leader_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941309d28e004b6aa8b4b81cbc4e46d4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35451 } } }
I20260925 05:10:34.417215 8491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:34.417868 8497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:34.418447 8497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:34.421731 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.415475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.3.6.190:35451)
0925 05:10:34.415691 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:34.415699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:34.415725 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:10:34.415878 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:34.415884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:34.415960 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:34.416101 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:34.416303 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.418144 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.419306 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.419952 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.420079 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.420960 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:34.420964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:34.420964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:34.421169 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:34.421221 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:34.421236 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:34.421311 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:34.421414 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 05:10:34.421500 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260925 05:10:34.421785 8497 catalog_manager.cc:1347] Generated new cluster ID: 5bf77e76d8f84cf5b4fc8ce894aad18c
I20260925 05:10:34.421836 8497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:34.422674 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:10:34.438670 8497 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:34.439419 8497 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:34.445277 8497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 941309d28e004b6aa8b4b81cbc4e46d4: Generated new TSK 0
I20260925 05:10:34.445505 8497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:10:34.461476 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.457600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.3.6.190:35451)
0925 05:10:34.457677 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:34.457680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:34.457789 (+ 109us) server_negotiation.cc:407] Connection header received
0925 05:10:34.457866 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:34.457867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:34.457878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:34.457944 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:10:34.458101 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.458610 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.459825 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.460022 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.460087 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.460279 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:10:34.461265 (+ 986us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:10:34.461398 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 05:10:34.461544 8490 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 05:10:34.461673 8515 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:34.457510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:35451 (local address 127.0.0.1:35920)
0925 05:10:34.457697 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:34.457710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:34.457787 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:34.457962 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:34.457963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:34.457965 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:10:34.458024 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:34.458027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.458690 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.458691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:34.459684 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:34.459688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.460097 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.460105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:34.460163 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.460169 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:10:34.461375 (+ 1206us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:10:34.461480 (+ 105us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.6.190:35451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":119,"threads_started":1}
I20260925 05:10:34.461899 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8419
E20260925 05:10:34.472874 8516 webserver.cc:533] Webserver: sslize failed:
E20260925 05:10:34.472990 8516 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T05:10:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T05:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:34Z Disabled control of system clock
I20260925 05:10:34.508539 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32851
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:32851
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/testchainca.cert with env {}
W20260925 05:10:34.591354 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:34.591583 8520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:34.591606 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:34.592949 8520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:34.593000 8520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:34.593014 8520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:34.593026 8520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:34.593039 8520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:10:34.593076 8520 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:10:34.593104 8520 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 05:10:34.594663 8520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:32851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:34.594937 8520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:34.595178 8520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:34.600301 8525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:34.600337 8526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:34.600308 8528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:34.600800 8520 server_base.cc:1034] running on GCE node
I20260925 05:10:34.601078 8520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:34.601425 8520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:34.602604 8520 hybrid_clock.cc:648] HybridClock initialized: now 1790313034602593 us; error 36 us; skew 500 ppm
I20260925 05:10:34.603950 8520 webserver.cc:460] Webserver started at http://127.3.6.190:44725/ using document root <none> and password file <none>
I20260925 05:10:34.604229 8520 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:34.604298 8520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:34.604440 8520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:34.605542 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "9d893746b7804c6886a3683d13b597ae"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "ccf178dd43e8ea7ec9a6c33cb47b11e3"
server_key_iv: "83ac8e85613b16a7a2f4e09f0cfd0013"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.605988 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "9d893746b7804c6886a3683d13b597ae"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "ccf178dd43e8ea7ec9a6c33cb47b11e3"
server_key_iv: "83ac8e85613b16a7a2f4e09f0cfd0013"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.607381 8520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:10:34.608259 8534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:34.608438 8520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:10:34.608548 8520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "9d893746b7804c6886a3683d13b597ae"
format_stamp: "Formatted at 2026-09-25 05:10:34 on dist-test-slave-1lvv"
server_key: "ccf178dd43e8ea7ec9a6c33cb47b11e3"
server_key_iv: "83ac8e85613b16a7a2f4e09f0cfd0013"
server_key_version: "encryptionkey@0"
I20260925 05:10:34.608624 8520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:34.618249 8520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:34.618625 8520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:34.618800 8520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:34.623188 8520 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:32851
I20260925 05:10:34.623220 8586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:32851 every 8 connection(s)
I20260925 05:10:34.623601 8520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:34.624693 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:34.627137 8587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae: Bootstrap starting.
I20260925 05:10:34.627900 8587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:34.628202 8587 log.cc:824] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:34.628973 8587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae: No bootstrap required, opened a new log
I20260925 05:10:34.630627 8587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d893746b7804c6886a3683d13b597ae" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32851 } }
I20260925 05:10:34.630784 8587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:34.630807 8587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d893746b7804c6886a3683d13b597ae, State: Initialized, Role: FOLLOWER
I20260925 05:10:34.631000 8587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d893746b7804c6886a3683d13b597ae" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32851 } }
I20260925 05:10:34.631084 8587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:34.631135 8587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:34.631194 8587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:34.631865 8587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d893746b7804c6886a3683d13b597ae" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32851 } }
I20260925 05:10:34.631979 8587 leader_election.cc:302] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d893746b7804c6886a3683d13b597ae; no voters:
I20260925 05:10:34.632180 8587 leader_election.cc:288] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:34.632239 8590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:34.632452 8587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:34.632463 8590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [term 1 LEADER]: Becoming Leader. State: Replica: 9d893746b7804c6886a3683d13b597ae, State: Running, Role: LEADER
I20260925 05:10:34.632581 8590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d893746b7804c6886a3683d13b597ae" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32851 } }
I20260925 05:10:34.633296 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8520
I20260925 05:10:34.633409 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:34.633625 3098 external_mini_cluster.cc:1428] Setting key e6db52f769c2c054e38ce9169e513bc9
I20260925 05:10:34.633740 8591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d893746b7804c6886a3683d13b597ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d893746b7804c6886a3683d13b597ae" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32851 } } }
I20260925 05:10:34.633828 8591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:34.634048 8605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:34.632987 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d893746b7804c6886a3683d13b597ae. Latest consensus state: current_term: 1 leader_uuid: "9d893746b7804c6886a3683d13b597ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d893746b7804c6886a3683d13b597ae" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32851 } } }
I20260925 05:10:34.634400 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:34.634763 8605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:34.636505 8605 catalog_manager.cc:1347] Generated new cluster ID: c5f6b81f7e4242b49ef92a1311f2e638
I20260925 05:10:34.636564 8605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:34.640815 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.634262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.3.6.190:32851)
0925 05:10:34.634457 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:34.634466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:34.635046 (+ 580us) server_negotiation.cc:407] Connection header received
0925 05:10:34.635401 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:34.635410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:34.635489 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:34.635654 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:34.635924 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.637270 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.638448 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.639103 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.639231 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.640055 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:34.640059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:34.640059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:10:34.640275 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:34.640318 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:10:34.640334 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:34.640399 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:34.640505 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 05:10:34.640584 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260925 05:10:34.641845 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:10:34.667299 8605 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:34.667940 8605 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:34.677652 8605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d893746b7804c6886a3683d13b597ae: Generated new TSK 0
I20260925 05:10:34.677891 8605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:10:34.713910 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:34.688172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.3.6.190:32851)
0925 05:10:34.688242 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:34.688245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:34.688389 (+ 144us) server_negotiation.cc:407] Connection header received
0925 05:10:34.688465 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:34.688466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:34.688477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:34.688541 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:10:34.688737 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.689225 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.690389 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.690570 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.690622 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.690842 (+ 220us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:10:34.713681 (+ 22839us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:10:34.713850 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59960: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 05:10:34.714006 8608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59960: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 05:10:34.714233 8616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:34.688108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32851 (local address 127.0.0.1:59960)
0925 05:10:34.688305 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:34.688318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:34.688392 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:34.688580 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:34.688581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:34.688584 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:10:34.688652 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:34.688656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.689327 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.689328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:34.690293 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:34.690296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:34.690664 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:34.690673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:34.690735 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:34.690741 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:10:34.713847 (+ 23106us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:10:34.714004 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.6.190:32851: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":111,"threads_started":1}
I20260925 05:10:34.714675 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8520
2026-09-25T05:10:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (232 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:34 dist-test-slave-1lvv krb5kdc[8628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:34 dist-test-slave-1lvv krb5kdc[8628](info): set up 2 sockets
Sep 25 05:10:34 dist-test-slave-1lvv krb5kdc[8628](info): commencing operation
krb5kdc: starting...
W20260925 05:10:35.948383 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:36.081137 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40577
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:40577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:36.164608 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:36.164899 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:36.164939 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:36.166184 8641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:36.166266 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:36.166291 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:36.166314 8641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:36.166343 8641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:36.167974 8641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:40577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:36.168308 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:36.168562 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:36.173712 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:36.173705 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:36.173705 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:36.173962 8641 server_base.cc:1034] running on GCE node
I20260925 05:10:36.174269 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:36.174590 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:36.175819 8641 hybrid_clock.cc:648] HybridClock initialized: now 1790313036175807 us; error 44 us; skew 500 ppm
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.182246 8641 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:36.182776 8641 webserver.cc:460] Webserver started at http://127.3.6.190:46543/ using document root <none> and password file <none>
I20260925 05:10:36.183019 8641 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:36.183096 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:36.183259 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:36.184509 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "6a985930e5d04161a5ea34576832522b"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "50055305ef7ba1355ad676ef222be841"
server_key_iv: "82bb5bbf2b3944bd584fb661e86163ea"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.184902 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "6a985930e5d04161a5ea34576832522b"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "50055305ef7ba1355ad676ef222be841"
server_key_iv: "82bb5bbf2b3944bd584fb661e86163ea"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.186451 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:36.187252 8656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.187443 8641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:36.187556 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "6a985930e5d04161a5ea34576832522b"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "50055305ef7ba1355ad676ef222be841"
server_key_iv: "82bb5bbf2b3944bd584fb661e86163ea"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.187635 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:36.202037 8641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:36.204906 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:36.205104 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:36.209443 8641 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:40577
I20260925 05:10:36.209484 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:40577 every 8 connection(s)
I20260925 05:10:36.209878 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:36.210824 8709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:36.213249 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b: Bootstrap starting.
I20260925 05:10:36.213829 8709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:36.214078 8709 log.cc:824] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:36.214787 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b: No bootstrap required, opened a new log
I20260925 05:10:36.216508 8709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a985930e5d04161a5ea34576832522b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40577 } }
I20260925 05:10:36.216665 8709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:36.216697 8709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a985930e5d04161a5ea34576832522b, State: Initialized, Role: FOLLOWER
I20260925 05:10:36.216809 8709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a985930e5d04161a5ea34576832522b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40577 } }
I20260925 05:10:36.216893 8709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:36.216943 8709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:36.217003 8709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:36.217132 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8641
I20260925 05:10:36.217248 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:36.217453 3098 external_mini_cluster.cc:1428] Setting key 7a2f792fc5518b1f70fc5cc50801c26b
I20260925 05:10:36.217805 8709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a985930e5d04161a5ea34576832522b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40577 } }
I20260925 05:10:36.217949 8709 leader_election.cc:302] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a985930e5d04161a5ea34576832522b; no voters:
I20260925 05:10:36.218196 8709 leader_election.cc:288] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:36.218441 8714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:36.218544 8709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:36.218606 8714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [term 1 LEADER]: Becoming Leader. State: Replica: 6a985930e5d04161a5ea34576832522b, State: Running, Role: LEADER
I20260925 05:10:36.218766 8714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a985930e5d04161a5ea34576832522b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40577 } }
I20260925 05:10:36.219213 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a985930e5d04161a5ea34576832522b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a985930e5d04161a5ea34576832522b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40577 } } }
I20260925 05:10:36.219235 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a985930e5d04161a5ea34576832522b. Latest consensus state: current_term: 1 leader_uuid: "6a985930e5d04161a5ea34576832522b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a985930e5d04161a5ea34576832522b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40577 } } }
I20260925 05:10:36.219311 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:36.219331 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:36.219517 8723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:36.220124 8723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:36.221688 8723 catalog_manager.cc:1347] Generated new cluster ID: 33edca20ca6c43f3aa4295bfa4e01237
I20260925 05:10:36.221738 8723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:36.229120 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.218004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.3.6.190:40577)
0925 05:10:36.218242 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.218254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.218295 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:10:36.218455 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.218463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.218520 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.219560 (+ 1040us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.219698 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.221023 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.222048 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.222475 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.222608 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.225077 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:36.225081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:36.225081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:36.225094 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:36.227441 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.227876 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.227886 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.227887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.227982 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.228206 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.228207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.228208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.228463 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:36.228650 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:36.228752 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 05:10:36.228904 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":122,"threads_started":1}
I20260925 05:10:36.245560 8723 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:36.246161 8723 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:36.251572 8723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a985930e5d04161a5ea34576832522b: Generated new TSK 0
I20260925 05:10:36.251871 8723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:36.284108 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40577
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:36.368827 8735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:36.369097 8735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:36.369138 8735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:36.370406 8735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:36.370491 8735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:36.370543 8735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:36.372143 8735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:36.372426 8735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:36.372677 8735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:36.378149 8741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:36.378180 8743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:36.378149 8740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:36.378185 8735 server_base.cc:1034] running on GCE node
I20260925 05:10:36.378712 8735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:36.379055 8735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:36.380256 8735 hybrid_clock.cc:648] HybridClock initialized: now 1790313036380225 us; error 53 us; skew 500 ppm
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.382992 8735 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:36.383586 8735 webserver.cc:460] Webserver started at http://127.3.6.129:42381/ using document root <none> and password file <none>
I20260925 05:10:36.383985 8735 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:36.384076 8735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:36.384261 8735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:36.385491 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "24fab0469476430e815e9f08ef82bd9f"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "97a524ec7dd693c23ddd692718b6cd4e"
server_key_iv: "0a2e0905550feccea47d008c8518e4bb"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.385933 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "24fab0469476430e815e9f08ef82bd9f"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "97a524ec7dd693c23ddd692718b6cd4e"
server_key_iv: "0a2e0905550feccea47d008c8518e4bb"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.387545 8735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:10:36.388552 8750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.388782 8735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:36.388906 8735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "24fab0469476430e815e9f08ef82bd9f"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "97a524ec7dd693c23ddd692718b6cd4e"
server_key_iv: "0a2e0905550feccea47d008c8518e4bb"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.388990 8735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:36.413314 8735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:36.415508 8735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:36.415699 8735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:36.415999 8735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:36.416358 8735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:36.416416 8735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.416461 8735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:36.416502 8735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.422906 8735 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:33669
I20260925 05:10:36.422987 8863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:33669 every 8 connection(s)
I20260925 05:10:36.423364 8735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:36.430487 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8735
I20260925 05:10:36.430609 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:36.430836 3098 external_mini_cluster.cc:1428] Setting key bd8f0ec657fcb9e817f7430d329ce764
I20260925 05:10:36.433221 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.424189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:47351 (local address 127.3.6.190:40577)
0925 05:10:36.424265 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.424267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.424479 (+ 212us) server_negotiation.cc:407] Connection header received
0925 05:10:36.425421 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.425422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.425437 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.425488 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.426091 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.426593 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.427589 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.427864 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.427919 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.430075 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:36.430077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:36.430078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:36.430088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:36.431817 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.432212 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.432213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.432213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.432277 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.432510 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.432511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.432511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.432658 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:36.432727 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:36.432978 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 05:10:36.433074 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:10:36.433282 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.424178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40577 (local address 127.3.6.129:47351)
0925 05:10:36.424354 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:36.424392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:36.425329 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:36.425552 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:36.425555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:36.425825 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:36.426018 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:36.426023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.426680 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.426681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:36.427506 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:36.427508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.427938 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.427939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:36.428044 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.428532 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:36.428540 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:36.429958 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:36.431891 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:36.431899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:36.431900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:36.432133 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:36.432344 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:36.432345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:36.432345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:36.432448 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:36.432725 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:36.432730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:36.432849 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 05:10:36.432934 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:10:36.433060 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260925 05:10:36.433815 8864 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40577
I20260925 05:10:36.433936 8864 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:36.434130 8864 heartbeater.cc:511] Master 127.3.6.190:40577 requested a full tablet report, sending...
I20260925 05:10:36.434806 8673 ts_manager.cc:194] Registered new tserver with Master: 24fab0469476430e815e9f08ef82bd9f (127.3.6.129:33669)
I20260925 05:10:36.435798 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:47351
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:36.459424 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40577
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:36.547966 8869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:36.548204 8869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:36.548231 8869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:36.549479 8869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:36.549527 8869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:36.549585 8869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:36.551169 8869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:36.551443 8869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:36.551712 8869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:36.557032 8874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:36.557032 8875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:36.557286 8869 server_base.cc:1034] running on GCE node
W20260925 05:10:36.558749 8877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:36.559036 8869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:36.559274 8869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:36.560456 8869 hybrid_clock.cc:648] HybridClock initialized: now 1790313036560444 us; error 45 us; skew 500 ppm
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.563023 8869 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:36.563459 8869 webserver.cc:460] Webserver started at http://127.3.6.130:44105/ using document root <none> and password file <none>
I20260925 05:10:36.563676 8869 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:36.563721 8869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:36.563885 8869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:36.564971 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "e884720d392d4adfb96f93e5c92c1716"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "c63788b0c9c6151c9f43b54d279aa96a"
server_key_iv: "bbb1c50a4ccb861678333907f7fea5f9"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.565285 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "e884720d392d4adfb96f93e5c92c1716"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "c63788b0c9c6151c9f43b54d279aa96a"
server_key_iv: "bbb1c50a4ccb861678333907f7fea5f9"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.566658 8869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:36.567456 8884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.567665 8869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:36.567801 8869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "e884720d392d4adfb96f93e5c92c1716"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "c63788b0c9c6151c9f43b54d279aa96a"
server_key_iv: "bbb1c50a4ccb861678333907f7fea5f9"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.567894 8869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:36.582137 8869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:36.584724 8869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:36.584862 8869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:36.585161 8869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:36.585458 8869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:36.585492 8869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.585516 8869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:36.585529 8869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.592273 8869 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:40753
I20260925 05:10:36.592361 8997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:40753 every 8 connection(s)
I20260925 05:10:36.592715 8869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:36.595229 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 8869
I20260925 05:10:36.595317 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:36.595518 3098 external_mini_cluster.cc:1428] Setting key ec1da29ae3ec3f36b5699f670db08340
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:36.603329 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.593484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:46783 (local address 127.3.6.190:40577)
0925 05:10:36.593537 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.593539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.593783 (+ 244us) server_negotiation.cc:407] Connection header received
0925 05:10:36.594760 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.594761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.594772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.594838 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.595593 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.596389 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.597471 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.597715 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.597794 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.600238 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:36.600240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:36.600240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:36.600249 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:36.601773 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.602254 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.602255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.602256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.602326 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.602608 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.602610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.602610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.602750 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:36.602831 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:36.603135 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 05:10:36.603232 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:10:36.603609 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.593451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40577 (local address 127.3.6.130:46783)
0925 05:10:36.593637 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:36.593682 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:36.594654 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:36.594901 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:36.594904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:36.595197 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:36.595483 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:36.595488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.596501 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.596503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:36.597365 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:36.597367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.597833 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.597835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:36.598021 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.598614 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:36.598622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:36.600112 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:36.601883 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:36.601893 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:36.601896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:36.602160 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:36.602401 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:36.602403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:36.602403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:36.602535 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:36.602822 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:36.602828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:36.602992 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 05:10:36.603103 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:10:36.603292 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260925 05:10:36.604204 8998 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40577
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
I20260925 05:10:36.604337 8998 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:36.604574 8998 heartbeater.cc:511] Master 127.3.6.190:40577 requested a full tablet report, sending...
I20260925 05:10:36.605001 8673 ts_manager.cc:194] Registered new tserver with Master: e884720d392d4adfb96f93e5c92c1716 (127.3.6.130:40753)
I20260925 05:10:36.605402 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:46783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:36.623157 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40577
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:36.708593 9003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:36.708830 9003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:36.708859 9003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:36.710088 9003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:36.710135 9003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:36.710165 9003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:36.711694 9003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:36.711982 9003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:36.712204 9003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:36.717639 9008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:36.717716 9003 server_base.cc:1034] running on GCE node
W20260925 05:10:36.717654 9009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:36.717584 9011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:36.718150 9003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:36.718458 9003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:36.719645 9003 hybrid_clock.cc:648] HybridClock initialized: now 1790313036719627 us; error 49 us; skew 500 ppm
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.722515 9003 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:36.722972 9003 webserver.cc:460] Webserver started at http://127.3.6.131:39975/ using document root <none> and password file <none>
I20260925 05:10:36.723201 9003 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:36.723246 9003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:36.723346 9003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:36.724484 9003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "2bcfcfbc0f7a4a16947aa161804d532f"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "8caee1326ca8009f1394cbf441fb6e14"
server_key_iv: "a5f0b4d363ce25f228ace6ddb03ceab2"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.724812 9003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "2bcfcfbc0f7a4a16947aa161804d532f"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "8caee1326ca8009f1394cbf441fb6e14"
server_key_iv: "a5f0b4d363ce25f228ace6ddb03ceab2"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.726240 9003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:36.727115 9018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.727385 9003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:36.727522 9003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "2bcfcfbc0f7a4a16947aa161804d532f"
format_stamp: "Formatted at 2026-09-25 05:10:36 on dist-test-slave-1lvv"
server_key: "8caee1326ca8009f1394cbf441fb6e14"
server_key_iv: "a5f0b4d363ce25f228ace6ddb03ceab2"
server_key_version: "encryptionkey@0"
I20260925 05:10:36.727612 9003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:36.752274 9003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:36.754709 9003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:36.754904 9003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:36.755169 9003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:36.755513 9003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:36.755573 9003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.755620 9003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:36.755646 9003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:36.762380 9003 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:37531
I20260925 05:10:36.762440 9131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:37531 every 8 connection(s)
I20260925 05:10:36.762825 9003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:36.769342 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9003
I20260925 05:10:36.769441 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:36.769672 3098 external_mini_cluster.cc:1428] Setting key a684cb1846822ab539bee1de6bd1443e
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:36.773764 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.763705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:43221 (local address 127.3.6.190:40577)
0925 05:10:36.763794 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.763797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.764058 (+ 261us) server_negotiation.cc:407] Connection header received
0925 05:10:36.765099 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.765101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.765116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.765181 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.765911 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.766420 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.767547 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.767841 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.767926 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.770587 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:36.770593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:36.770593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:36.770602 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:36.772231 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.772702 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.772704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.772704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.772768 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.773054 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.773055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.773055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.773180 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:36.773252 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:36.773572 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 05:10:36.773663 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:36.773998 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.763667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40577 (local address 127.3.6.131:43221)
0925 05:10:36.763895 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:36.763935 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:36.764964 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:36.765267 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:36.765271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:36.765582 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:36.765793 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:36.765798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.766533 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.766534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:36.767430 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:36.767432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.767940 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.767941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:36.768099 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.768650 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:36.768659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:36.770454 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:36.772328 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:36.772339 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:36.772342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:36.772609 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:36.772836 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:36.772837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:36.772838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:36.772976 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:36.773258 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:36.773266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:36.773426 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 05:10:36.773548 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 05:10:36.773732 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":118,"threads_started":1}
I20260925 05:10:36.774537 9132 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40577
I20260925 05:10:36.774677 9132 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:36.774879 9132 heartbeater.cc:511] Master 127.3.6.190:40577 requested a full tablet report, sending...
I20260925 05:10:36.775298 8673 ts_manager.cc:194] Registered new tserver with Master: 2bcfcfbc0f7a4a16947aa161804d532f (127.3.6.131:37531)
I20260925 05:10:36.775655 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:43221
I20260925 05:10:36.782152 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/test-admin.keytab.
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.801132 3098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 05:10:36.803023 9139 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:36.802168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.3.6.129:33669)
0925 05:10:36.802365 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.802370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.802391 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:10:36.802418 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.802420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.802435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.802506 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.802915 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56854: BlockingRecv error: recv got EOF from 127.0.0.1:56854 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":127,"threads_started":1}
W20260925 05:10:36.803104 3098 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.804550 3098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.805740 3098 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:36.814692 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.806377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.3.6.129:33669)
0925 05:10:36.806464 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.806466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.806479 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:10:36.806544 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.806545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.806557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.806615 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.807038 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.807896 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.808668 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.809088 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.809174 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.811749 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:36.811822 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:36.811822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:36.811831 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:36.813584 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.813937 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.813940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.813940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.814001 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.814193 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.814194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.814194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.814333 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:36.814454 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:36.814519 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 05:10:36.814609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 05:10:36.816918 9139 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:36.816285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.3.6.129:33669)
0925 05:10:36.816343 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.816345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.816403 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:10:36.816481 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.816482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.816493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.816542 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.816858 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56880: BlockingRecv error: recv got EOF from 127.0.0.1:56880 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 05:10:36.816984 3098 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.818120 3098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.819221 3098 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:36.826504 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.819908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.3.6.129:33669)
0925 05:10:36.819975 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.819977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.820107 (+ 130us) server_negotiation.cc:407] Connection header received
0925 05:10:36.820155 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.820155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.820167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.820222 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.820624 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.821126 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.821816 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.822008 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.822061 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.824234 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:36.824235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:36.824236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:36.824243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:36.825461 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.825809 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.825810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.825810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.825864 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.826052 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.826053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.826054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.826190 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:36.826274 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:36.826359 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 05:10:36.826435 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 05:10:36.828244 9139 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:36.827604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.3.6.129:33669)
0925 05:10:36.827666 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.827669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.827742 (+ 73us) server_negotiation.cc:407] Connection header received
0925 05:10:36.827844 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.827845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.827855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.827908 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.828192 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56894: BlockingRecv error: recv got EOF from 127.0.0.1:56894 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 05:10:36.828322 3098 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790313008371521-3098-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.829382 3098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:36.830384 3098 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:10:36.837929 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:36.830967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.3.6.129:33669)
0925 05:10:36.831032 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:36.831034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:36.831090 (+ 56us) server_negotiation.cc:407] Connection header received
0925 05:10:36.831158 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:36.831159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:36.831169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:36.831244 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:36.831688 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.832261 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.832996 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:36.833203 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:36.833290 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:36.835420 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:36.835422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:36.835422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:36.835431 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:36.836774 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.837127 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.837129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.837130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.837194 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:36.837399 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:36.837400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:36.837401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:36.837511 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:36.837619 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:36.837734 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 05:10:36.837819 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:10:36.838589 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8735
I20260925 05:10:36.843413 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8869
I20260925 05:10:36.848592 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9003
I20260925 05:10:36.853603 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 8641
2026-09-25T05:10:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1956 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[9152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[9152](info): set up 2 sockets
Sep 25 05:10:36 dist-test-slave-1lvv krb5kdc[9152](info): commencing operation
krb5kdc: starting...
W20260925 05:10:37.900563 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:37 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab.
Entry for principal oryx/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab.
I20260925 05:10:38.031744 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40397
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab
--principal=oryx/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:38.116925 9165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:38.117261 9165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:38.117290 9165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:38.118536 9165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:38.118592 9165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:38.118613 9165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:38.118626 9165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:38.118638 9165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:38.120324 9165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab
--principal=oryx/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:38.120615 9165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:38.120869 9165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:38.126189 9173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.126237 9171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.126171 9170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:38.126565 9165 server_base.cc:1034] running on GCE node
I20260925 05:10:38.126858 9165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:38.127238 9165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:38.128407 9165 hybrid_clock.cc:648] HybridClock initialized: now 1790313038128388 us; error 46 us; skew 500 ppm
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:38.131108 9165 init.cc:377] Logged in from keytab as oryx/127.3.6.190@KRBTEST.COM (short username oryx)
I20260925 05:10:38.131623 9165 webserver.cc:460] Webserver started at http://127.3.6.190:39599/ using document root <none> and password file <none>
I20260925 05:10:38.131953 9165 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:38.132035 9165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:38.132205 9165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:38.133387 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "494ef92f61e24303ad5538bc97340890"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "de39777fcccd1c60b3c6271dd7bed655"
server_key_iv: "e5bd848415b80154a59f36b08e5f8e7d"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.133888 9165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "494ef92f61e24303ad5538bc97340890"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "de39777fcccd1c60b3c6271dd7bed655"
server_key_iv: "e5bd848415b80154a59f36b08e5f8e7d"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.135457 9165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:38.136382 9180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.136576 9165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:10:38.136693 9165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "494ef92f61e24303ad5538bc97340890"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "de39777fcccd1c60b3c6271dd7bed655"
server_key_iv: "e5bd848415b80154a59f36b08e5f8e7d"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.136785 9165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:38.146346 9165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:38.148679 9165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:38.148838 9165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:38.153194 9165 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:40397
I20260925 05:10:38.153228 9232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:40397 every 8 connection(s)
I20260925 05:10:38.153637 9165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:38.154788 9233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:38.157299 9233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: Bootstrap starting.
I20260925 05:10:38.157730 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9165
I20260925 05:10:38.157831 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:38.157971 9233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:38.158042 3098 external_mini_cluster.cc:1428] Setting key f4135d55e6e7364a99ec0d37fd94fc7f
I20260925 05:10:38.158339 9233 log.cc:824] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:38.159272 9233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: No bootstrap required, opened a new log
I20260925 05:10:38.161804 9233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } }
I20260925 05:10:38.162000 9233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:38.162058 9233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494ef92f61e24303ad5538bc97340890, State: Initialized, Role: FOLLOWER
I20260925 05:10:38.162220 9233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } }
I20260925 05:10:38.162340 9233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:38.162389 9233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:38.162446 9233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:38.163189 9233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } }
I20260925 05:10:38.163307 9233 leader_election.cc:302] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 494ef92f61e24303ad5538bc97340890; no voters:
I20260925 05:10:38.163492 9233 leader_election.cc:288] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:38.163553 9238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:38.163853 9233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:38.163848 9238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 1 LEADER]: Becoming Leader. State: Replica: 494ef92f61e24303ad5538bc97340890, State: Running, Role: LEADER
I20260925 05:10:38.164682 9238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } }
I20260925 05:10:38.165372 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } }
I20260925 05:10:38.165802 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:38.165512 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494ef92f61e24303ad5538bc97340890. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } }
I20260925 05:10:38.166047 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
W20260925 05:10:38.166374 9252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:38.166493 9252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:10:38.166567 9254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:38.166800 9254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:38.168819 9254 catalog_manager.cc:1347] Generated new cluster ID: 01264a14d0714e6b96ec9098c4721ede
I20260925 05:10:38.168893 9254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:38.170908 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.158664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.3.6.190:40397)
0925 05:10:38.158882 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.158916 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.158953 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:10:38.159597 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.159607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.159677 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.159866 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.160488 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.161948 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.162878 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.163554 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.163681 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.167025 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.167029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.167029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.167041 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.169298 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.169737 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.169740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.169740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.169835 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.170035 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.170036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.170036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.170279 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.170400 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.170526 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.170645 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260925 05:10:38.187932 9254 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:38.188644 9254 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:38.192765 9254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: Generated new TSK 0
I20260925 05:10:38.193003 9254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab.
Entry for principal oryx/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab.
I20260925 05:10:38.227195 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40397
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab
--principal=oryx/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:38.313330 9259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:38.313556 9259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:38.313591 9259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:38.314719 9259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:38.314762 9259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:38.314790 9259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:38.316221 9259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab
--principal=oryx/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:38.316423 9259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:38.316630 9259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:38.321935 9264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.321981 9265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.321934 9267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:38.322332 9259 server_base.cc:1034] running on GCE node
I20260925 05:10:38.322593 9259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:38.322816 9259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:38.323984 9259 hybrid_clock.cc:648] HybridClock initialized: now 1790313038323968 us; error 35 us; skew 500 ppm
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:38.326563 9259 init.cc:377] Logged in from keytab as oryx/127.3.6.129@KRBTEST.COM (short username oryx)
I20260925 05:10:38.327040 9259 webserver.cc:460] Webserver started at http://127.3.6.129:33057/ using document root <none> and password file <none>
I20260925 05:10:38.327314 9259 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:38.327385 9259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:38.327539 9259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:38.328704 9259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "8402cb6315f041fc9846bda9b25af18b"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "6f1cf8302830051ae1237fa6ea775656"
server_key_iv: "982a5175cbf9ee2308b3c2fb378bac99"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.329093 9259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "8402cb6315f041fc9846bda9b25af18b"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "6f1cf8302830051ae1237fa6ea775656"
server_key_iv: "982a5175cbf9ee2308b3c2fb378bac99"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.330552 9259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:38.331362 9274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.331554 9259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:38.331650 9259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "8402cb6315f041fc9846bda9b25af18b"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "6f1cf8302830051ae1237fa6ea775656"
server_key_iv: "982a5175cbf9ee2308b3c2fb378bac99"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.331750 9259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:38.343007 9259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:38.345052 9259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:38.345235 9259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:38.345489 9259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:38.345829 9259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:38.345882 9259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.345924 9259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:38.345968 9259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.352278 9259 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:36383
I20260925 05:10:38.352319 9387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:36383 every 8 connection(s)
I20260925 05:10:38.352710 9259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:38.352907 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9259
I20260925 05:10:38.353024 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:38.353272 3098 external_mini_cluster.cc:1428] Setting key 4536d21a021a2f30cb09558cc05d7c7c
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.6.130@KRBTEST.COM; defaulting to no policy
I20260925 05:10:38.364207 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.353521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:48475 (local address 127.3.6.190:40397)
0925 05:10:38.353573 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.353575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.353814 (+ 239us) server_negotiation.cc:407] Connection header received
0925 05:10:38.354555 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.354555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.354565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.354606 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.356039 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.356550 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.358168 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.358493 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.358555 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.361083 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.361084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.361085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.361093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.362608 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.363023 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.363024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.363024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.363087 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.363348 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.363349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.363349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.363520 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.363588 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.363998 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.364098 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 05:10:38.364454 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.353504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:48475)
0925 05:10:38.353677 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.353727 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.354442 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.354692 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.354697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.355575 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.355921 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.355928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.356661 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.356662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.358073 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.358076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.358596 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.358597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.358757 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.359299 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.359310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.360980 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.362703 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.362710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.362712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.362941 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.363158 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.363159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.363159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.363277 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.363590 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.363597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.363752 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.363960 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.364149 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260925 05:10:38.365114 9388 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.130@KRBTEST.COM" created.
I20260925 05:10:38.365232 9388 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:38.365439 9388 heartbeater.cc:511] Master 127.3.6.190:40397 requested a full tablet report, sending...
I20260925 05:10:38.366024 9197 ts_manager.cc:194] Registered new tserver with Master: 8402cb6315f041fc9846bda9b25af18b (127.3.6.129:36383)
I20260925 05:10:38.366662 9197 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:48475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab.
Entry for principal oryx/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab.
I20260925 05:10:38.382542 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40397
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab
--principal=oryx/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:38.466394 9393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:38.466647 9393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:38.466688 9393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:38.467902 9393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:38.467979 9393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:38.468030 9393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:38.469564 9393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab
--principal=oryx/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:38.469864 9393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:38.470127 9393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:38.475330 9401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.475337 9398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.475350 9399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:38.475693 9393 server_base.cc:1034] running on GCE node
I20260925 05:10:38.475934 9393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:38.476143 9393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:38.477314 9393 hybrid_clock.cc:648] HybridClock initialized: now 1790313038477302 us; error 44 us; skew 500 ppm
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:38.479895 9393 init.cc:377] Logged in from keytab as oryx/127.3.6.130@KRBTEST.COM (short username oryx)
I20260925 05:10:38.480340 9393 webserver.cc:460] Webserver started at http://127.3.6.130:44575/ using document root <none> and password file <none>
I20260925 05:10:38.480589 9393 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:38.480659 9393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:38.480813 9393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:38.481876 9393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "93740a3f9acf4c9f8b167d7c98ec132d"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "e581c0bce75224d2ad8f1e0c513c81d1"
server_key_iv: "9a386e6909cd179926aa60268b130b4e"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.482249 9393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "93740a3f9acf4c9f8b167d7c98ec132d"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "e581c0bce75224d2ad8f1e0c513c81d1"
server_key_iv: "9a386e6909cd179926aa60268b130b4e"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.483711 9393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:38.484611 9408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.484790 9393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:38.484901 9393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "93740a3f9acf4c9f8b167d7c98ec132d"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "e581c0bce75224d2ad8f1e0c513c81d1"
server_key_iv: "9a386e6909cd179926aa60268b130b4e"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.484978 9393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:38.515262 9393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:38.517825 9393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:38.518035 9393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:38.518370 9393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:38.518710 9393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:38.518746 9393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.518770 9393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:38.518817 9393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.524699 9393 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:45535
I20260925 05:10:38.524789 9521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:45535 every 8 connection(s)
I20260925 05:10:38.525100 9393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:38.528388 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9393
I20260925 05:10:38.528498 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:38.528713 3098 external_mini_cluster.cc:1428] Setting key cfabea96cd780ef887a534267b16abfb
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:38.535645 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.525894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:46233 (local address 127.3.6.190:40397)
0925 05:10:38.525968 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.525970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.526198 (+ 228us) server_negotiation.cc:407] Connection header received
0925 05:10:38.526735 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.526736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.526749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.526809 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.527868 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.528537 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.529639 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.529928 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.530009 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.532216 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.532218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.532218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.532226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.534003 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.534416 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.534418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.534418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.534489 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.534856 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.534857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.534858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.535036 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.535123 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.535396 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.535500 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:10:38.535820 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.525877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:46233)
0925 05:10:38.526055 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.526100 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.526620 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.526887 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.526891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.527497 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.527704 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.527709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.528654 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.528655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.529546 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.529548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.530025 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.530026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.530144 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.530685 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.530693 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.532124 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.534101 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.534107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.534109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.534337 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.534638 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.534639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.534641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.534777 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.535114 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.535119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.535251 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.535345 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.535485 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260925 05:10:38.536321 9522 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40397
I20260925 05:10:38.536441 9522 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:38.536656 9522 heartbeater.cc:511] Master 127.3.6.190:40397 requested a full tablet report, sending...
I20260925 05:10:38.537096 9197 ts_manager.cc:194] Registered new tserver with Master: 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535)
I20260925 05:10:38.537634 9197 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:46233
WARNING: no policy specified for oryx/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab.
Entry for principal oryx/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab.
I20260925 05:10:38.559087 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40397
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab
--principal=oryx/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:38.642947 9527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:38.643163 9527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:38.643195 9527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:38.644536 9527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:38.644582 9527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:38.644610 9527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:38.646056 9527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab
--principal=oryx/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:38.646297 9527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:38.646546 9527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:38.651743 9535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.651743 9532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.651721 9533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:38.651958 9527 server_base.cc:1034] running on GCE node
I20260925 05:10:38.652192 9527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:38.652420 9527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:38.653585 9527 hybrid_clock.cc:648] HybridClock initialized: now 1790313038653546 us; error 41 us; skew 500 ppm
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:38.656124 9527 init.cc:377] Logged in from keytab as oryx/127.3.6.131@KRBTEST.COM (short username oryx)
I20260925 05:10:38.656620 9527 webserver.cc:460] Webserver started at http://127.3.6.131:44747/ using document root <none> and password file <none>
I20260925 05:10:38.656853 9527 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:38.656898 9527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:38.657071 9527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:38.658174 9527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "563cc985ed2940f0900bddbb1e8be419"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "d8aa6af43dc8d4bb5fb7a3a81e1dfe17"
server_key_iv: "448d0955831bbd7bb52dbd6c1da01894"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.658543 9527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "563cc985ed2940f0900bddbb1e8be419"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "d8aa6af43dc8d4bb5fb7a3a81e1dfe17"
server_key_iv: "448d0955831bbd7bb52dbd6c1da01894"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.660136 9527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:38.660908 9542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.661084 9527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:38.661180 9527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "563cc985ed2940f0900bddbb1e8be419"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "d8aa6af43dc8d4bb5fb7a3a81e1dfe17"
server_key_iv: "448d0955831bbd7bb52dbd6c1da01894"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.661238 9527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:38.681156 9527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:38.683120 9527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:38.683252 9527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:38.683452 9527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:38.683843 9527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:38.683884 9527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.683909 9527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:38.683954 9527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.689905 9527 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:34929
I20260925 05:10:38.689949 9655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:34929 every 8 connection(s)
I20260925 05:10:38.690286 9527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:38.695041 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9527
I20260925 05:10:38.695158 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:38.695354 3098 external_mini_cluster.cc:1428] Setting key f28040de17e2fe91759d89823437d43d
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:38.699747 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.691041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:38271 (local address 127.3.6.190:40397)
0925 05:10:38.691116 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.691118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.691309 (+ 191us) server_negotiation.cc:407] Connection header received
0925 05:10:38.691888 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.691889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.691900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.691965 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.693071 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.693598 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.694588 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.694763 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.694813 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.697133 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.697134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.697134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.697140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.698320 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.698708 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.698708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.698709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.698765 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.699106 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.699107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.699107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.699273 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.699340 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.699610 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.699680 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:10:38.700021 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.691005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:38271)
0925 05:10:38.691187 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.691229 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.691734 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.692041 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.692045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.692717 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.692983 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.692989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.693693 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.693694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.694512 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.694515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.694835 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.694836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.694935 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.695504 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.695512 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.697051 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.698390 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.698396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.698398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.698617 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.698827 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.698828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.698828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.698929 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.699345 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.699350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.699471 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.699562 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.699690 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260925 05:10:38.700527 9656 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40397
I20260925 05:10:38.700621 9656 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:38.700793 9656 heartbeater.cc:511] Master 127.3.6.190:40397 requested a full tablet report, sending...
I20260925 05:10:38.701175 9197 ts_manager.cc:194] Registered new tserver with Master: 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929)
I20260925 05:10:38.701591 9197 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:38271
I20260925 05:10:38.708107 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.6.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.189.keytab.
Entry for principal oryx/127.3.6.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.189.keytab.
I20260925 05:10:38.739310 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:40491
--webserver_interface=127.3.6.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.189.keytab
--principal=oryx/127.3.6.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:40397,127.3.6.189:40491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:38.823971 9661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:38.824229 9661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:38.824270 9661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:38.825534 9661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:38.825627 9661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:38.825654 9661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:38.825685 9661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:38.825717 9661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:38.827299 9661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:40397,127.3.6.189:40491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.189.keytab
--principal=oryx/127.3.6.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.6.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
W20260925 05:10:38.827464 9661 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.6.190:40397,127.3.6.189:40491'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 05:10:38.827643 9661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:38.827946 9661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:38.832998 9667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.833020 9669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.833006 9666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:38.833081 9661 server_base.cc:1034] running on GCE node
I20260925 05:10:38.833432 9661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:38.833802 9661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:38.834972 9661 hybrid_clock.cc:648] HybridClock initialized: now 1790313038834949 us; error 37 us; skew 500 ppm
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:38.837759 9661 init.cc:377] Logged in from keytab as oryx/127.3.6.189@KRBTEST.COM (short username oryx)
I20260925 05:10:38.838275 9661 webserver.cc:460] Webserver started at http://127.3.6.189:37421/ using document root <none> and password file <none>
I20260925 05:10:38.838534 9661 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:38.838605 9661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:38.838765 9661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:38.840014 9661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/instance:
uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "db0bde0b295c57384fae280265d869d9"
server_key_iv: "ff4723883efa48b2b77f62418dfb83c4"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.840389 9661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal/instance:
uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "db0bde0b295c57384fae280265d869d9"
server_key_iv: "ff4723883efa48b2b77f62418dfb83c4"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.841868 9661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:38.842638 9676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.842813 9661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:38.842922 9661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "db0bde0b295c57384fae280265d869d9"
server_key_iv: "ff4723883efa48b2b77f62418dfb83c4"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.842999 9661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:38.853492 9661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:38.856016 9661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:38.856191 9661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:38.860910 9661 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.189:40491
I20260925 05:10:38.860944 9728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.189:40491 every 8 connection(s)
I20260925 05:10:38.861330 9661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
I20260925 05:10:38.862362 9729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:38.864250 9729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:10:38.865025 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9661
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.189@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.6.189@KRBTEST.COM
I20260925 05:10:38.874859 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.865118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.3.6.190:40397)
0925 05:10:38.865208 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.865211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.865457 (+ 246us) server_negotiation.cc:407] Connection header received
0925 05:10:38.866120 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.866121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.866135 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.866192 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.866776 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.867319 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.868377 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.868585 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.868676 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.871215 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.871217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.871217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.871228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.873024 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.873636 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.873637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.873655 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.873742 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.874114 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.874115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.874115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.874361 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.874415 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.874648 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.874758 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:10:38.875073 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.865102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:40626)
0925 05:10:38.865313 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.865356 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.866004 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.866210 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.866212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.866506 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.866671 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.866676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.867435 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.867436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.868275 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.868278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.868686 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.868687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.868822 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.869388 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.869397 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.871112 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.873175 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.873182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.873184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.873560 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.873850 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.873851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.873852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.874007 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.874428 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.874433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.874517 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.874620 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.874799 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260925 05:10:38.875290 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.865650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.3.6.189:40491)
0925 05:10:38.865800 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.865805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.865839 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:10:38.866002 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.866008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.866027 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.866133 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.866775 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.867618 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.868450 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.868790 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.868886 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.871704 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.871712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.871713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.871720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.873607 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.873982 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.873985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.873986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.874041 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.874264 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.874264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.874269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.874527 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.874719 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.874793 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.874854 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260925 05:10:38.875710 9729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.189@KRBTEST.COM for oryx/127.3.6.189@KRBTEST.COM
I20260925 05:10:38.881752 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.876031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.0.0.1:43544)
0925 05:10:38.876059 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.876068 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.876099 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.876205 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.876205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.876316 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.876365 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.876367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.876930 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.876931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.877401 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.877402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.877744 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.877744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.877792 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.878231 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.878236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.879375 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.880914 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.880915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.880915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.881068 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.881265 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.881266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.881266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.881313 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.881489 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.881490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.881522 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.881568 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.881645 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 05:10:38.881791 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.876058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.3.6.189:40491)
0925 05:10:38.876115 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.876117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.876126 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:38.876151 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.876151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.876159 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.876203 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.876406 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.876871 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.877469 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.877680 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.877734 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.879446 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.879446 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.879447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.879452 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.880830 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.881149 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.881150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.881150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.881201 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.881351 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.881351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.881352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.881441 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.881485 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.881595 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.881690 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":38}
I20260925 05:10:38.883307 9729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Bootstrap starting.
I20260925 05:10:38.884267 9729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:38.884517 9729 log.cc:824] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:38.885288 9729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: No bootstrap required, opened a new log
I20260925 05:10:38.886722 9729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } }
I20260925 05:10:38.886865 9729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:38.886888 9729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ef226760b44a5a899e1faf6b9b0fc8, State: Initialized, Role: FOLLOWER
I20260925 05:10:38.886984 9729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } }
I20260925 05:10:38.887285 9735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } } }
I20260925 05:10:38.887370 9729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:38.887374 9735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:10:38.889189 9746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:38.889261 9746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:10:38.890962 9661 master_runner.cc:428] Detected that this master 92ef226760b44a5a899e1faf6b9b0fc8 is joining an existing cluster
I20260925 05:10:38.891006 9661 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.6.189:40491
I20260925 05:10:38.898685 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.892192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.0.0.1:43556)
0925 05:10:38.892357 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.892368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.892451 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.892575 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.892576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.892856 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.892937 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.892939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.893577 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.893578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.894158 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.894159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.894534 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.894534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.894602 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.895093 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.895098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.895410 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.897531 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.897532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.897532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.897705 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.897929 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.897930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.897930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.897992 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.898239 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.898240 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.898278 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.898379 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.898544 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":6,"thread_start_us":113,"threads_started":1}
I20260925 05:10:38.898695 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.892192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.3.6.189:40491)
0925 05:10:38.892264 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.892267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.892436 (+ 169us) server_negotiation.cc:407] Connection header received
0925 05:10:38.892508 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.892509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.892518 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.892569 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.893019 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.893498 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.894256 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.894452 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.894532 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.895515 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.895516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.895517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.895523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.897445 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.897798 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.897799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.897800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.897853 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.898057 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.898058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.898058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.898172 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.898248 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.898400 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.898599 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":42}
I20260925 05:10:38.900027 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.892202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.3.6.190:40397)
0925 05:10:38.892308 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.892310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.892597 (+ 287us) server_negotiation.cc:407] Connection header received
0925 05:10:38.892706 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.892707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.892719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.892778 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.893177 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.893947 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.895030 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.895331 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.895375 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.896507 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.896509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.896509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.896518 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.898184 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.898633 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.898635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.898635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.898694 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.899436 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.899437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.899437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.899554 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.899612 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.899830 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.899931 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:10:38.900028 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.892374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:40628)
0925 05:10:38.892516 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.892527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.892624 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.892795 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.892796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.893000 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.893075 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.893077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.894049 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.894050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.894927 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.894930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.895453 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.895454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.895515 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.895964 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.895969 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.896415 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.898279 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.898280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.898280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.898534 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.899301 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.899302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.899302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.899375 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.899628 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.899630 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.899677 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.899788 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.899912 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.190@KRBTEST.COM for oryx/127.3.6.189@KRBTEST.COM
I20260925 05:10:38.910380 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.902715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.3.6.189:40491)
0925 05:10:38.902758 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.902760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.902911 (+ 151us) server_negotiation.cc:407] Connection header received
0925 05:10:38.902967 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.902968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.902980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.903047 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.903457 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.904269 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.905049 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.905358 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.905433 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.907492 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.907494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.907494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.907502 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.909093 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.909606 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.909607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.909607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.909667 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.909880 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.909881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.909881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.909987 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.910038 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.910249 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.910315 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:38.910459 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.902627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.0.0.1:43570)
0925 05:10:38.902803 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.902840 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.902907 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.903050 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.903052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.903327 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.903383 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.903385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.904349 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.904350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.904965 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.904967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.905438 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.905438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.905500 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.906008 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.906012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.907405 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.909217 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.909227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.909229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.909514 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.909741 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.909742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.909742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.909816 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.910048 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.910056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.910119 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.910212 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.910347 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260925 05:10:38.911108 9197 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.6.189:40491: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
member_type: NON_VOTER
last_known_addr {
host: "127.3.6.189"
port: 40491
}
attrs {
promote: true
}
}
dest_uuid: "494ef92f61e24303ad5538bc97340890"
cas_config_opid_index: -1
I20260925 05:10:38.911448 9197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } }
I20260925 05:10:38.911993 9756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } }
I20260925 05:10:38.912400 9703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:38.913035 9756 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.6.189:40491
I20260925 05:10:38.913283 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } }
I20260925 05:10:38.913376 9703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Refusing update from remote peer 494ef92f61e24303ad5538bc97340890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 05:10:38.913475 9661 master.cc:537] Master@127.3.6.189:40491 shutting down...
I20260925 05:10:38.913763 9756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:38.914642 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
W20260925 05:10:38.915439 9183 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 05:10:38.916110 9183 consensus_peers.cc:487] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 -> Peer 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189:40491): Couldn't send request to peer 92ef226760b44a5a899e1faf6b9b0fc8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 05:10:38.916883 9661 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:38.916991 9661 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:38.917047 9661 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: stopping tablet replica
I20260925 05:10:38.930802 9661 master.cc:552] Master@127.3.6.189:40491 shutdown complete.
I20260925 05:10:38.930876 9661 master_runner.cc:305] Clearing existing system tablet
I20260925 05:10:38.931020 9661 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:38.931074 9661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:38.931947 9661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:38.932500 9760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.932679 9661 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:38.932745 9661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.932771 9661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "db0bde0b295c57384fae280265d869d9"
server_key_iv: "ff4723883efa48b2b77f62418dfb83c4"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.933605 9661 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:38.935935 9661 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:10:38.936021 9661 log.cc:1196] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 05:10:38.936242 9661 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Deleting consensus metadata
I20260925 05:10:38.936372 9661 master_runner.cc:315] Copying system tablet from 127.3.6.190:40397
I20260925 05:10:38.937043 9661 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: tablet copy: Beginning tablet copy session from remote peer at address 127.3.6.190:40397
I20260925 05:10:38.943645 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.937463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.3.6.190:40397)
0925 05:10:38.937518 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.937520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.937582 (+ 62us) server_negotiation.cc:407] Connection header received
0925 05:10:38.937658 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.937659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.937680 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.937734 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.938117 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.938606 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.939734 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.939993 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.940054 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.941130 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.941132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.941132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.941138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.942497 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.942965 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.942966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.942966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.943018 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.943217 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.943218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.943218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.943318 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.943369 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.943522 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.943579 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:10:38.943737 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.937352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:40632)
0925 05:10:38.937494 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:38.937506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:38.937594 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:38.937753 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:38.937754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:38.937967 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:38.938042 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.938045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.938685 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.938685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.939614 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:38.939617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.940069 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.940070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:38.940143 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.940643 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:38.940648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:38.941048 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:38.942652 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.942653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.942654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.942885 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.943082 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:38.943083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:38.943083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:38.943153 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:38.943380 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:38.943381 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:38.943429 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:10:38.943507 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:10:38.943620 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":95,"threads_started":1}
I20260925 05:10:38.944345 9217 tablet_copy_service.cc:140] P 494ef92f61e24303ad5538bc97340890: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 92ef226760b44a5a899e1faf6b9b0fc8 ({username='oryx', principal='oryx/127.3.6.189@KRBTEST.COM'} at 127.0.0.1:40632)
I20260925 05:10:38.944456 9217 tablet_copy_service.cc:161] P 494ef92f61e24303ad5538bc97340890: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 92ef226760b44a5a899e1faf6b9b0fc8 at {username='oryx', principal='oryx/127.3.6.189@KRBTEST.COM'} at 127.0.0.1:40632: session id = 92ef226760b44a5a899e1faf6b9b0fc8-00000000000000000000000000000000
I20260925 05:10:38.945133 9217 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:10:38.945852 9661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:10:38.947218 9661 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: tablet copy: Starting download of 0 data blocks...
I20260925 05:10:38.947392 9661 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: tablet copy: Starting download of 1 WAL segments...
I20260925 05:10:38.948414 9661 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:10:38.949426 9217 tablet_copy_service.cc:342] P 494ef92f61e24303ad5538bc97340890: Request end of tablet copy session 92ef226760b44a5a899e1faf6b9b0fc8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.6.189@KRBTEST.COM'} at 127.0.0.1:40632
I20260925 05:10:38.949491 9217 tablet_copy_service.cc:433] P 494ef92f61e24303ad5538bc97340890: ending tablet copy session 92ef226760b44a5a899e1faf6b9b0fc8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 92ef226760b44a5a899e1faf6b9b0fc8
I20260925 05:10:38.952116 9661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:38.953424 9767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.953524 9770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:38.953526 9768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:38.953804 9661 server_base.cc:1034] running on GCE node
I20260925 05:10:38.953891 9661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:38.954056 9661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:38.955180 9661 hybrid_clock.cc:648] HybridClock initialized: now 1790313038955163 us; error 34 us; skew 500 ppm
Sep 25 05:10:38 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:38.956633 9661 init.cc:377] Logged in from keytab as oryx/127.3.6.189@KRBTEST.COM (short username oryx)
I20260925 05:10:38.956950 9661 webserver.cc:460] Webserver started at http://127.3.6.189:36201/ using document root <none> and password file <none>
I20260925 05:10:38.957098 9661 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:38.957140 9661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:38.957800 9661 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:38.958408 9777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:38.958552 9661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:38.958600 9661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "db0bde0b295c57384fae280265d869d9"
server_key_iv: "ff4723883efa48b2b77f62418dfb83c4"
server_key_version: "encryptionkey@0"
I20260925 05:10:38.958698 9661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:38.967851 9661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:38.969563 9661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:38.972841 9661 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.189:40491
I20260925 05:10:38.973129 9829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.189:40491 every 8 connection(s)
I20260925 05:10:38.973304 9661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
I20260925 05:10:38.974076 9830 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:10:38.974519 9830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Bootstrap starting.
I20260925 05:10:38.977471 9830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:10:38.977746 9830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: Bootstrap complete.
I20260925 05:10:38.978018 9830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } }
I20260925 05:10:38.978176 9830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92ef226760b44a5a899e1faf6b9b0fc8, State: Initialized, Role: LEARNER
I20260925 05:10:38.978245 9830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } }
I20260925 05:10:38.978456 9830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:38.979807 9834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } }
I20260925 05:10:38.980011 9834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:10:38.980064 9845 catalog_manager.cc:1259] Loaded cluster ID: 01264a14d0714e6b96ec9098c4721ede
I20260925 05:10:38.980175 9845 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: loading cluster ID for follower catalog manager: success
I20260925 05:10:38.981316 9845 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: acquiring CA information for follower catalog manager: success
I20260925 05:10:38.981662 9845 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:10:38.985060 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:38.977096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.3.6.189:40491)
0925 05:10:38.977246 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:38.977248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:38.977258 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:38.977284 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:38.977285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:38.977296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:38.977391 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:38.978152 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.979162 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.980576 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:38.980772 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:38.980862 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:38.982225 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:38.982227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:38.982227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:38.982235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:38.983736 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.984152 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.984154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.984154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.984232 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:38.984445 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:38.984446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:38.984447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:38.984575 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:38.984678 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:38.984846 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:10:38.984954 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.3.6.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.188.keytab.
Entry for principal oryx/127.3.6.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.188.keytab.
I20260925 05:10:39.014508 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:40615
--webserver_interface=127.3.6.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.188.keytab
--principal=oryx/127.3.6.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:39.100920 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:39.101152 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:39.101176 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:39.102391 9848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:39.102440 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:39.102455 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:39.102468 9848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:39.102522 9848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:39.104133 9848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.188.keytab
--principal=oryx/127.3.6.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:40615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.6.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:39.104408 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:39.104656 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:39.109544 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:39.109597 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.109769 9848 server_base.cc:1034] running on GCE node
W20260925 05:10:39.109544 9856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.110217 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:39.110529 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:39.111698 9848 hybrid_clock.cc:648] HybridClock initialized: now 1790313039111691 us; error 38 us; skew 500 ppm
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:39.114392 9848 init.cc:377] Logged in from keytab as oryx/127.3.6.188@KRBTEST.COM (short username oryx)
I20260925 05:10:39.114850 9848 webserver.cc:460] Webserver started at http://127.3.6.188:45951/ using document root <none> and password file <none>
I20260925 05:10:39.115089 9848 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:39.115160 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:39.115314 9848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:39.116477 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/instance:
uuid: "c0d6b213f8fc456a94f2933328964a6a"
format_stamp: "Formatted at 2026-09-25 05:10:39 on dist-test-slave-1lvv"
server_key: "5669b0c0d080f6a79bbbf625399fcc6f"
server_key_iv: "1038503d0be5fc69f1220c594e1ab959"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.116863 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal/instance:
uuid: "c0d6b213f8fc456a94f2933328964a6a"
format_stamp: "Formatted at 2026-09-25 05:10:39 on dist-test-slave-1lvv"
server_key: "5669b0c0d080f6a79bbbf625399fcc6f"
server_key_iv: "1038503d0be5fc69f1220c594e1ab959"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.118525 9848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:39.119594 9863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.119793 9848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:39.119890 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
uuid: "c0d6b213f8fc456a94f2933328964a6a"
format_stamp: "Formatted at 2026-09-25 05:10:39 on dist-test-slave-1lvv"
server_key: "5669b0c0d080f6a79bbbf625399fcc6f"
server_key_iv: "1038503d0be5fc69f1220c594e1ab959"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.120020 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:39.134651 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:39.137209 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:39.137379 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:39.141692 9848 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.188:40615
I20260925 05:10:39.141808 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.188:40615 every 8 connection(s)
I20260925 05:10:39.142184 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
I20260925 05:10:39.143221 9916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:39.144877 9916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:10:39.150616 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9848
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.188@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:39.154811 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.145780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.3.6.190:40397)
0925 05:10:39.145836 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.145838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.145920 (+ 82us) server_negotiation.cc:407] Connection header received
0925 05:10:39.146416 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.146417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.146427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.146492 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.147112 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.147760 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.148792 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.148983 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.149042 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.151543 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.151545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.151545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.151554 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.153244 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.153780 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.153782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.153783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.153847 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.154127 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.154128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.154129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.154268 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.154315 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.154623 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.154726 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:10:39.155012 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.145602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:40646)
0925 05:10:39.145799 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.145844 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.146323 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.146559 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.146563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.146823 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.147029 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.147034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.147881 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.147882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.148706 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.148708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.149062 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.149063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.149165 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.149682 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.149690 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.151431 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.153361 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.153367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.153370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.153683 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.153918 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.153919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.153919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.154042 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.154356 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.154361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.154485 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.154599 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.154761 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":117,"threads_started":1}
I20260925 05:10:39.156250 9916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.6.188@KRBTEST.COM
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.188@KRBTEST.COM for oryx/127.3.6.189@KRBTEST.COM
I20260925 05:10:39.160822 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.151292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.3.6.188:40615)
0925 05:10:39.151480 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.151484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.151506 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:10:39.151526 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.151528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.151539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.151629 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.152438 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.153206 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.154266 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.154736 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.155112 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.157590 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.157596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.157596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.157602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.159428 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.159924 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.159927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.159927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.160008 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.160234 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.160235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.160236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.160379 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.160563 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.160615 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.160707 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260925 05:10:39.163343 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.156595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.0.0.1:43608)
0925 05:10:39.157123 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.157133 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.157175 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.157303 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.157303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.157411 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.157457 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.157458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.158143 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.158145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.159002 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.159004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.159352 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.159352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.159435 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.159823 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.159828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.161232 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.162508 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.162509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.162510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.162668 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.162831 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.162832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.162832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.162878 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.163080 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.163081 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.163121 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.163202 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.163279 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511}
I20260925 05:10:39.163367 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.156629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.3.6.189:40491)
0925 05:10:39.156676 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.156678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.157185 (+ 507us) server_negotiation.cc:407] Connection header received
0925 05:10:39.157216 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.157217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.157229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.157302 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.157535 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.158047 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.159079 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.159276 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.159331 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.161317 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.161318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.161318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.161324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.162441 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.162729 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.162730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.162730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.162777 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.162915 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.162916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.162916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.163005 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.163051 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.163233 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.163305 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:39.163849 9916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.188@KRBTEST.COM for oryx/127.3.6.188@KRBTEST.COM
I20260925 05:10:39.170279 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.164125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.0.0.1:41516)
0925 05:10:39.164175 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.164186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.164223 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.164371 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.164372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.164482 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.164515 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.164516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.165312 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.165313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.165802 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.165804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.166120 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.166121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.166155 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.166499 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.166503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.167714 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.169393 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.169394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.169394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.169591 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.169776 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.169777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.169777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.169829 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.170014 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.170015 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.170051 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.170101 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.170186 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:10:39.170329 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.164173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.3.6.188:40615)
0925 05:10:39.164244 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.164246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.164256 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:39.164273 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.164274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.164283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.164364 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.164590 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.165243 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.165875 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.166063 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.166115 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.167873 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.167875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.167875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.167882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.169306 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.169658 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.169658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.169659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.169719 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.169867 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.169868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.169868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.169966 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.170009 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.170117 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.170231 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":51}
I20260925 05:10:39.171911 9916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Bootstrap starting.
I20260925 05:10:39.172811 9916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:39.173111 9916 log.cc:824] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:39.173947 9916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: No bootstrap required, opened a new log
I20260925 05:10:39.175423 9916 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } }
I20260925 05:10:39.175563 9916 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:39.175585 9916 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d6b213f8fc456a94f2933328964a6a, State: Initialized, Role: FOLLOWER
I20260925 05:10:39.175805 9916 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } }
I20260925 05:10:39.176167 9916 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:39.176137 9921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } } }
I20260925 05:10:39.176267 9921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:10:39.177901 9932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:10:39.177964 9932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:10:39.180487 9848 master_runner.cc:428] Detected that this master c0d6b213f8fc456a94f2933328964a6a is joining an existing cluster
I20260925 05:10:39.180532 9848 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.6.188:40615
I20260925 05:10:39.188665 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.182123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.3.6.188:40615)
0925 05:10:39.182160 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.182162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.182254 (+ 92us) server_negotiation.cc:407] Connection header received
0925 05:10:39.182338 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.182339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.182346 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.182374 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.182752 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.183401 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184497 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.184780 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184826 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.186002 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.186003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.186004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.186010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.187420 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.187867 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.187868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.187868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.187923 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.188122 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.188123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.188123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.188241 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.188316 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.188470 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.188555 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:39.188747 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.181781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.0.0.1:41532)
0925 05:10:39.182202 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.182210 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.182277 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.182423 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.182424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.182610 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.182663 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.182664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.183480 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.183481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.184400 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.184403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.184920 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.184981 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.185513 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.185518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.185890 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.187518 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.187518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.187519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.187745 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.187999 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.188000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.188001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.188059 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.188306 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.188307 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.188364 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.188449 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.188608 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":70,"thread_start_us":64,"threads_started":1}
I20260925 05:10:39.212219 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9259
I20260925 05:10:39.212349 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.181758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.3.6.189:40491)
0925 05:10:39.181954 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.181956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.181967 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:39.182002 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.182003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.182013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.182061 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.182510 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.183297 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184515 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.184810 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184852 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.209118 (+ 24266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.209120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.209120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.209129 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.210936 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211402 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.211403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.211404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.211464 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211683 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.211684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.211684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.211859 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.211905 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.212121 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.212226 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260925 05:10:39.212714 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.181584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.0.0.1:43616)
0925 05:10:39.181898 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.181907 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.181949 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.182063 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.182063 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.182334 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.182417 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.182419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.183389 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.183389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.184422 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.184424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.184922 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.184981 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.208533 (+ 23552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.208539 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.208979 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.211043 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.211045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.211318 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.211542 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.211543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.211604 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.211953 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.211954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.212002 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.212428 (+ 426us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.212555 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":85,"thread_start_us":86,"threads_started":1}
I20260925 05:10:39.213688 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.181591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.3.6.190:40397)
0925 05:10:39.181641 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.181644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.181768 (+ 124us) server_negotiation.cc:407] Connection header received
0925 05:10:39.181886 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.181888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.181902 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.181965 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.182481 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.183265 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184580 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.184886 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.184931 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.209196 (+ 24265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.209199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.209199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.209208 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.211085 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211522 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.211523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.211523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.211617 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211845 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.211846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.211847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.211989 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.212034 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.212234 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.213589 (+ 1355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:10:39.214053 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.181526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:40658)
0925 05:10:39.181705 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.181716 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.181796 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.182019 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.182020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.182297 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.182378 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.182381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.183368 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.183368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.184458 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.184461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.185015 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.185016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.185081 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.208463 (+ 23382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.208470 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.209084 (+ 614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.211183 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.211184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.211430 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.211693 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.211694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.211695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.211773 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.212083 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.212084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.212128 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.213827 (+ 1699us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.213946 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":88,"thread_start_us":101,"threads_started":1}
I20260925 05:10:39.218575 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:36383
--local_ip_for_outbound_sockets=127.3.6.129
--tserver_master_addrs=127.3.6.190:40397
--webserver_port=33057
--webserver_interface=127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab
--principal=oryx/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313038, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.190@KRBTEST.COM for oryx/127.3.6.188@KRBTEST.COM
I20260925 05:10:39.229101 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.220042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.3.6.188:40615)
0925 05:10:39.220733 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.220736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.220746 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:39.220799 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.220800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.220812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.220883 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.221273 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.221811 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.222901 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.223265 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.223338 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.225920 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.225922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.225922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.225931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.227699 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.228201 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.228202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.228202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.228264 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.228507 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.228507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.228513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.228653 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.228724 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.228922 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.229001 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662}
I20260925 05:10:39.229161 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.219984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.0.0.1:41534)
0925 05:10:39.220042 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.220054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.220094 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.220911 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.220913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.221117 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.221177 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.221180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.221896 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.221897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.222811 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.222814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.223351 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.223352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.223429 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.224005 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.224010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.225816 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.227853 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.227855 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.227855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.228118 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.228346 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.228347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.228348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.228430 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.228732 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.228734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.228793 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.228886 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.229012 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:10:39.230105 9197 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.6.188:40615: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a"
member_type: NON_VOTER
last_known_addr {
host: "127.3.6.188"
port: 40615
}
attrs {
promote: true
}
}
dest_uuid: "494ef92f61e24303ad5538bc97340890"
cas_config_opid_index: 5
I20260925 05:10:39.230420 9197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } }
I20260925 05:10:39.231053 9756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c0d6b213f8fc456a94f2933328964a6a (127.3.6.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.232172 9756 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.6.188:40615
I20260925 05:10:39.232280 9942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.232438 9942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:39.232841 9848 master.cc:537] Master@127.3.6.188:40615 shutting down...
I20260925 05:10:39.233145 9890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:39.234133 9890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Refusing update from remote peer 494ef92f61e24303ad5538bc97340890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:10:39.235536 9942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:10:39.237483 9181 consensus_peers.cc:487] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 -> Peer c0d6b213f8fc456a94f2933328964a6a (127.3.6.188:40615): Couldn't send request to peer c0d6b213f8fc456a94f2933328964a6a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 05:10:39.238315 9848 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:39.238413 9848 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:39.238459 9848 tablet_replica.cc:330] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: stopping tablet replica
I20260925 05:10:39.240892 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.232526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.3.6.189:40491)
0925 05:10:39.232568 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.232570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.236175 (+ 3605us) server_negotiation.cc:407] Connection header received
0925 05:10:39.236293 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.236294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.236318 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.236359 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:39.236568 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.237357 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.239319 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.240084 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.240137 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.240218 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.240771 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.240820 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:10:39.241140 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.232429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.0.0.1:43626)
0925 05:10:39.236104 (+ 3675us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.236118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.236212 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.236445 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.236446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.236447 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:39.236499 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.236501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.237907 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.237909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.239191 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.239194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.240575 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.240576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.240650 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.240681 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.240980 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.241032 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3651}
I20260925 05:10:39.241425 9804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 LEARNER]: Refusing update from remote peer 494ef92f61e24303ad5538bc97340890: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:10:39.241647 9942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 05:10:39.242194 9804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c0d6b213f8fc456a94f2933328964a6a (127.3.6.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.243332 9834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494ef92f61e24303ad5538bc97340890. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.243407 9834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:10:39.243438 9834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: NON_VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: true } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.243474 9834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:10:39.243844 9942 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: attempting to promote NON_VOTER 92ef226760b44a5a899e1faf6b9b0fc8 to VOTER
I20260925 05:10:39.244051 9942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } }
I20260925 05:10:39.244748 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.244601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.0.0.1:41540)
0925 05:10:39.244665 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.244719 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:40615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:10:39.244773 9804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 LEARNER]: Refusing update from remote peer 494ef92f61e24303ad5538bc97340890: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 05:10:39.244959 9942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 05:10:39.245184 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.245097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.0.0.1:41548)
0925 05:10:39.245134 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.245158 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:40615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 05:10:39.245247 9181 consensus_peers.cc:487] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 -> Peer c0d6b213f8fc456a94f2933328964a6a (127.3.6.188:40615): Couldn't send request to peer c0d6b213f8fc456a94f2933328964a6a. Status: Network error: Client connection negotiation failed: client connection to 127.3.6.188:40615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:10:39.245651 9756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.246652 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.246757 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:39.247067 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.247129 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:39.247416 9804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.248427 9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.248518 9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:39.251885 9848 master.cc:552] Master@127.3.6.188:40615 shutdown complete.
I20260925 05:10:39.251940 9848 master_runner.cc:305] Clearing existing system tablet
I20260925 05:10:39.252074 9848 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:39.252122 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:39.252883 9848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:39.253454 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.253655 9848 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:39.253724 9848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:39.253747 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
uuid: "c0d6b213f8fc456a94f2933328964a6a"
format_stamp: "Formatted at 2026-09-25 05:10:39 on dist-test-slave-1lvv"
server_key: "5669b0c0d080f6a79bbbf625399fcc6f"
server_key_iv: "1038503d0be5fc69f1220c594e1ab959"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.254436 9848 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:39.255868 9848 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:10:39.255940 9848 log.cc:1196] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 05:10:39.256150 9848 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Deleting consensus metadata
I20260925 05:10:39.256299 9848 master_runner.cc:315] Copying system tablet from 127.3.6.190:40397
I20260925 05:10:39.256914 9848 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: tablet copy: Beginning tablet copy session from remote peer at address 127.3.6.190:40397
I20260925 05:10:39.264767 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.257315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.3.6.190:40397)
0925 05:10:39.257383 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.257385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.257528 (+ 143us) server_negotiation.cc:407] Connection header received
0925 05:10:39.257579 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.257580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.257592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.257674 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.258107 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.258848 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.260048 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.260328 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.260414 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.261622 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.261623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.261624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.261632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.263373 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.263825 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.263827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.263827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.263896 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.264139 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.264141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.264141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.264290 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.264337 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.264584 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.264672 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:10:39.264768 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.257299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:40672)
0925 05:10:39.257441 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.257452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.257517 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.257695 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.257696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.257900 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.257967 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.257969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.258949 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.258951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.259950 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.259953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.260433 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.260434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.260502 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.261077 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.261082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.261516 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.263461 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.263462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.263463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.263705 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.263975 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.263976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.263976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.264050 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.264418 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.264419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.264472 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.264542 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.264655 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260925 05:10:39.265498 9217 tablet_copy_service.cc:140] P 494ef92f61e24303ad5538bc97340890: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c0d6b213f8fc456a94f2933328964a6a ({username='oryx', principal='oryx/127.3.6.188@KRBTEST.COM'} at 127.0.0.1:40672)
I20260925 05:10:39.265568 9217 tablet_copy_service.cc:161] P 494ef92f61e24303ad5538bc97340890: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c0d6b213f8fc456a94f2933328964a6a at {username='oryx', principal='oryx/127.3.6.188@KRBTEST.COM'} at 127.0.0.1:40672: session id = c0d6b213f8fc456a94f2933328964a6a-00000000000000000000000000000000
I20260925 05:10:39.266153 9217 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:10:39.266726 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:10:39.268232 9848 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: tablet copy: Starting download of 0 data blocks...
I20260925 05:10:39.268393 9848 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: tablet copy: Starting download of 1 WAL segments...
I20260925 05:10:39.269222 9848 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:10:39.270208 9217 tablet_copy_service.cc:342] P 494ef92f61e24303ad5538bc97340890: Request end of tablet copy session c0d6b213f8fc456a94f2933328964a6a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.6.188@KRBTEST.COM'} at 127.0.0.1:40672
I20260925 05:10:39.270272 9217 tablet_copy_service.cc:433] P 494ef92f61e24303ad5538bc97340890: ending tablet copy session c0d6b213f8fc456a94f2933328964a6a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c0d6b213f8fc456a94f2933328964a6a
I20260925 05:10:39.272996 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:39.274361 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:39.276690 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:39.277017 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.277603 9848 server_base.cc:1034] running on GCE node
I20260925 05:10:39.277714 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:39.277855 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:39.278971 9848 hybrid_clock.cc:648] HybridClock initialized: now 1790313039278962 us; error 29 us; skew 500 ppm
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:39.280696 9848 init.cc:377] Logged in from keytab as oryx/127.3.6.188@KRBTEST.COM (short username oryx)
I20260925 05:10:39.280982 9848 webserver.cc:460] Webserver started at http://127.3.6.188:40123/ using document root <none> and password file <none>
I20260925 05:10:39.281111 9848 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:39.281145 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:39.281976 9848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:39.282625 9964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.282748 9848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:39.282790 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
uuid: "c0d6b213f8fc456a94f2933328964a6a"
format_stamp: "Formatted at 2026-09-25 05:10:39 on dist-test-slave-1lvv"
server_key: "5669b0c0d080f6a79bbbf625399fcc6f"
server_key_iv: "1038503d0be5fc69f1220c594e1ab959"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.282857 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:39.289588 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:39.291623 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:39.294895 9848 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.188:40615
I20260925 05:10:39.295318 10016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.188:40615 every 8 connection(s)
I20260925 05:10:39.295370 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
I20260925 05:10:39.296417 10017 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:10:39.297008 10017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Bootstrap starting.
I20260925 05:10:39.300199 10017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:10:39.300462 10017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Bootstrap complete.
I20260925 05:10:39.300675 10017 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } }
I20260925 05:10:39.300801 10017 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0d6b213f8fc456a94f2933328964a6a, State: Initialized, Role: LEARNER
I20260925 05:10:39.300851 10017 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } }
I20260925 05:10:39.301046 10017 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:39.301071 10020 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.301296 10020 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: This master's current role is: LEARNER
I20260925 05:10:39.302532 10031 catalog_manager.cc:1259] Loaded cluster ID: 01264a14d0714e6b96ec9098c4721ede
I20260925 05:10:39.303258 10031 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: loading cluster ID for follower catalog manager: success
I20260925 05:10:39.304432 10031 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: acquiring CA information for follower catalog manager: success
I20260925 05:10:39.304765 10031 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:10:39.329687 9941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:39.329955 9941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:39.330001 9941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:39.331218 9941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:39.331297 9941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:39.331351 9941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:39.332989 9941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab
--principal=oryx/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:36383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=33057
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:39.333308 9941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:39.333604 9941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:39.339684 10037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.339843 9941 server_base.cc:1034] running on GCE node
W20260925 05:10:39.339967 10039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:39.339978 10036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.340328 9941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:39.340631 9941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:39.341794 9941 hybrid_clock.cc:648] HybridClock initialized: now 1790313039341774 us; error 35 us; skew 500 ppm
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:39.344647 9941 init.cc:377] Logged in from keytab as oryx/127.3.6.129@KRBTEST.COM (short username oryx)
I20260925 05:10:39.345135 9941 webserver.cc:460] Webserver started at http://127.3.6.129:33057/ using document root <none> and password file <none>
I20260925 05:10:39.345407 9941 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:39.345516 9941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:39.346954 9941 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:39.347633 10046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.347846 9941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:39.347963 9941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "8402cb6315f041fc9846bda9b25af18b"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "6f1cf8302830051ae1237fa6ea775656"
server_key_iv: "982a5175cbf9ee2308b3c2fb378bac99"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.348364 9941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:39.360663 9941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:39.362649 9941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:39.362840 9941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:39.363112 9941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:39.363575 9941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:39.363636 9941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.363687 9941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:39.363735 9941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.371001 9941 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:36383
I20260925 05:10:39.371045 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:36383 every 8 connection(s)
I20260925 05:10:39.371512 9941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:39.375965 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 9941
I20260925 05:10:39.376158 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9393
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.189@KRBTEST.COM
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:39.383816 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:45535
--local_ip_for_outbound_sockets=127.3.6.130
--tserver_master_addrs=127.3.6.190:40397
--webserver_port=44575
--webserver_interface=127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab
--principal=oryx/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:39.384407 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.372423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:45655 (local address 127.3.6.190:40397)
0925 05:10:39.372508 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.372512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.372757 (+ 245us) server_negotiation.cc:407] Connection header received
0925 05:10:39.374054 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.374055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.374067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.374118 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.374844 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.375381 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.377619 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.377914 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.377983 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.380814 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.380816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.380816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.380825 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.382529 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.383322 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.383323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.383324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.383375 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.383616 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.383617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.383618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.383827 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.383872 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.384230 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.384327 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:10:39.384505 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.372362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:44765 (local address 127.3.6.189:40491)
0925 05:10:39.372482 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.372484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.372757 (+ 273us) server_negotiation.cc:407] Connection header received
0925 05:10:39.373901 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.373903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.373918 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.373980 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.374900 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.375380 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.376787 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.377091 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.377139 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.380830 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.380832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.380832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.380841 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.382442 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.383137 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.383139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.383139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.383225 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.383552 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.383553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.383553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.383727 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.383831 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.384243 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.384349 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 05:10:39.385149 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.372388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:45655)
0925 05:10:39.372610 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.372657 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.373768 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.374138 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.374139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.374506 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.374745 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.374751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.375500 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.375501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.377515 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.377518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.378044 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.378046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.378125 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.378646 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.378651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.380701 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.382610 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.382620 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.382623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.383045 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.383448 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.383449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.383450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.383530 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.383962 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.383963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.384067 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.384628 (+ 561us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.384949 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1499,"thread_start_us":138,"threads_started":1}
I20260925 05:10:39.385155 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.372388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.3.6.129:44765)
0925 05:10:39.372692 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.372701 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.373768 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.374062 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.374067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.374522 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.374745 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.374752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.375507 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.375508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.376674 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.376677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.377237 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.377238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.377420 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.378109 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.378121 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.380716 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.382609 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.382620 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.382623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.383051 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.383304 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.383305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.383306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.383460 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.383837 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.383846 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.384067 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.384609 (+ 542us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.384865 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":37,"thread_start_us":139,"threads_started":1}
I20260925 05:10:39.385696 10161 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40397
I20260925 05:10:39.385823 10161 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.385902 10162 heartbeater.cc:347] Connected to a master server at 127.3.6.189:40491
I20260925 05:10:39.385987 10162 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.386063 10161 heartbeater.cc:511] Master 127.3.6.190:40397 requested a full tablet report, sending...
I20260925 05:10:39.386072 10162 heartbeater.cc:511] Master 127.3.6.189:40491 requested a full tablet report, sending...
I20260925 05:10:39.386615 9197 ts_manager.cc:194] Re-registered known tserver with Master: 8402cb6315f041fc9846bda9b25af18b (127.3.6.129:36383)
I20260925 05:10:39.386675 9794 ts_manager.cc:194] Registered new tserver with Master: 8402cb6315f041fc9846bda9b25af18b (127.3.6.129:36383)
I20260925 05:10:39.387142 9197 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:45655
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.188@KRBTEST.COM
I20260925 05:10:39.393285 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.372821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:36377 (local address 127.3.6.188:40615)
0925 05:10:39.372964 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.372966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.372976 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:39.373972 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.373973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.373985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.374071 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.375512 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.386566 (+ 11054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.387969 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.388203 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.388323 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.390356 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.390357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.390358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.390374 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.392020 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.392389 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.392391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.392391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.392455 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.392685 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.392685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.392686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.392809 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.392889 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.393066 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.393169 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260925 05:10:39.393544 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.372746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.3.6.129:36377)
0925 05:10:39.372869 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.372880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.373795 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.374077 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.374077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.374501 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.374745 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.374751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.386684 (+ 11933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.386685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.387370 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.387373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.388322 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.388324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.388400 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.388985 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.388989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.390245 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.392110 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.392112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.392113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.392298 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.392546 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.392547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.392548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.392612 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.392881 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.392883 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.392945 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.393039 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.393170 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260925 05:10:39.394043 10160 heartbeater.cc:347] Connected to a master server at 127.3.6.188:40615
I20260925 05:10:39.394119 10160 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.394227 10160 heartbeater.cc:511] Master 127.3.6.188:40615 requested a full tablet report, sending...
I20260925 05:10:39.394709 9981 ts_manager.cc:194] Registered new tserver with Master: 8402cb6315f041fc9846bda9b25af18b (127.3.6.129:36383)
W20260925 05:10:39.472945 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:39.473193 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:39.473219 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:39.474351 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:39.474390 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:39.474417 10168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:39.475970 10168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab
--principal=oryx/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:45535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=44575
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:39.476291 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:39.476511 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:39.482038 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:39.482050 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.482298 10168 server_base.cc:1034] running on GCE node
W20260925 05:10:39.482050 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.482546 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:39.482759 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:39.483933 10168 hybrid_clock.cc:648] HybridClock initialized: now 1790313039483901 us; error 48 us; skew 500 ppm
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:39.486687 10168 init.cc:377] Logged in from keytab as oryx/127.3.6.130@KRBTEST.COM (short username oryx)
I20260925 05:10:39.487244 10168 webserver.cc:460] Webserver started at http://127.3.6.130:44575/ using document root <none> and password file <none>
I20260925 05:10:39.487488 10168 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:39.487577 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:39.489113 10168 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:39.489775 10183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.490123 10168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:39.490237 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "93740a3f9acf4c9f8b167d7c98ec132d"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "e581c0bce75224d2ad8f1e0c513c81d1"
server_key_iv: "9a386e6909cd179926aa60268b130b4e"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.490618 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:39.508481 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:39.510666 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:39.510834 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:39.511101 10168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:39.511431 10168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:39.511488 10168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.511531 10168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:39.511559 10168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.517704 10168 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:45535
I20260925 05:10:39.517755 10296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:45535 every 8 connection(s)
I20260925 05:10:39.518168 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:39.520659 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 10168
I20260925 05:10:39.520856 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9527
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.189@KRBTEST.COM
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.188@KRBTEST.COM
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:39.528198 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:34929
--local_ip_for_outbound_sockets=127.3.6.131
--tserver_master_addrs=127.3.6.190:40397
--webserver_port=44747
--webserver_interface=127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab
--principal=oryx/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:39.530719 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.518984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:60503 (local address 127.3.6.189:40491)
0925 05:10:39.519262 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.519265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.519425 (+ 160us) server_negotiation.cc:407] Connection header received
0925 05:10:39.520062 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.520064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.520076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.520108 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.521007 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.521753 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.523252 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.523550 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.523589 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.527497 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.527499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.527499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.527507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.529196 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529618 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.529619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.529619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.529675 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529908 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.529909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.529910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.530053 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.530102 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.530490 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.530585 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260925 05:10:39.530720 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.518985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:57885 (local address 127.3.6.188:40615)
0925 05:10:39.519034 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.519037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.519391 (+ 354us) server_negotiation.cc:407] Connection header received
0925 05:10:39.520017 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.520018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.520036 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.520091 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.521686 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.522523 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.523748 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.524162 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.524217 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.527130 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.527132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.527132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.527141 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.528938 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529497 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.529499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.529499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.529560 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529870 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.529871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.529871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.530017 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.530079 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.530497 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.530593 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:10:39.531339 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.518995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.3.6.130:57885)
0925 05:10:39.519196 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.519234 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.519908 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.520165 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.520169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.520578 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.521577 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.521580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.522628 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.522629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.523643 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.523646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.524295 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.524296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.524376 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.524821 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.524827 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.527017 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.529045 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.529055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.529402 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.529640 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.529641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.529777 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.530134 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.530141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.530311 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.530834 (+ 523us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.531053 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":116,"thread_start_us":99,"threads_started":1}
I20260925 05:10:39.531343 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.518995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.3.6.130:60503)
0925 05:10:39.519356 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.519368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.519927 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.520182 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.520183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.520594 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.520889 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.520896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.521885 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.521887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.523157 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.523159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.523680 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.523681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.523836 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.524429 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.524440 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.527416 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.529291 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.529292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.529535 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.529771 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.529771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.529772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.529836 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.530152 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.530153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.530313 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.530849 (+ 536us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.531167 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":166,"thread_start_us":87,"threads_started":1}
I20260925 05:10:39.531860 10299 heartbeater.cc:347] Connected to a master server at 127.3.6.189:40491
I20260925 05:10:39.531957 10299 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.532037 10297 heartbeater.cc:347] Connected to a master server at 127.3.6.188:40615
I20260925 05:10:39.532074 10297 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.532168 10299 heartbeater.cc:511] Master 127.3.6.189:40491 requested a full tablet report, sending...
I20260925 05:10:39.532167 10297 heartbeater.cc:511] Master 127.3.6.188:40615 requested a full tablet report, sending...
I20260925 05:10:39.532608 9981 ts_manager.cc:194] Registered new tserver with Master: 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535)
I20260925 05:10:39.532701 9794 ts_manager.cc:194] Registered new tserver with Master: 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535)
I20260925 05:10:39.533573 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.519438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:60559 (local address 127.3.6.190:40397)
0925 05:10:39.519488 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.519490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.519500 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:39.520056 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.520058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.520073 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.520108 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.522004 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.522746 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.523711 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.524020 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.524409 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.528860 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.528861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.528862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.528870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.530624 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.531382 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.531384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.531384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.531446 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.533038 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.533039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.533039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.533158 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.533237 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.533407 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.533485 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:10:39.533586 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.519319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:60559)
0925 05:10:39.519434 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.519442 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.519930 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.520181 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.520182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.520675 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.520889 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.520896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.522842 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.522843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.523618 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.523621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.524497 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.524497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.524839 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.525243 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.525249 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.527706 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.530708 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.530709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.530709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.531286 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.532781 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.532781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.532782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.532964 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.533241 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.533242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.533282 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.533367 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.533476 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260925 05:10:39.534296 10298 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40397
I20260925 05:10:39.534363 10298 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.534446 10298 heartbeater.cc:511] Master 127.3.6.190:40397 requested a full tablet report, sending...
I20260925 05:10:39.534719 9197 ts_manager.cc:194] Re-registered known tserver with Master: 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535)
I20260925 05:10:39.535122 9197 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:60559
W20260925 05:10:39.615839 10304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:39.616068 10304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:39.616093 10304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:39.617259 10304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:39.617308 10304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:39.617337 10304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:39.618840 10304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab
--principal=oryx/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:34929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=44747
--tserver_master_addrs=127.3.6.190:40397,127.3.6.189:40491,127.3.6.188:40615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:39.619068 10304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:39.619359 10304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:39.624599 10309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:39.624687 10312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:39.624610 10310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:39.624970 10304 server_base.cc:1034] running on GCE node
I20260925 05:10:39.625200 10304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:39.625484 10304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:39.626654 10304 hybrid_clock.cc:648] HybridClock initialized: now 1790313039626634 us; error 36 us; skew 500 ppm
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:39.629271 10304 init.cc:377] Logged in from keytab as oryx/127.3.6.131@KRBTEST.COM (short username oryx)
I20260925 05:10:39.629807 10304 webserver.cc:460] Webserver started at http://127.3.6.131:44747/ using document root <none> and password file <none>
I20260925 05:10:39.630131 10304 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:39.630229 10304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:39.634720 10304 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.002s
I20260925 05:10:39.635607 10319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.635906 10304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:39.636009 10304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "563cc985ed2940f0900bddbb1e8be419"
format_stamp: "Formatted at 2026-09-25 05:10:38 on dist-test-slave-1lvv"
server_key: "d8aa6af43dc8d4bb5fb7a3a81e1dfe17"
server_key_iv: "448d0955831bbd7bb52dbd6c1da01894"
server_key_version: "encryptionkey@0"
I20260925 05:10:39.636551 10304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:39.653051 10304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:39.655324 10304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:39.655505 10304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:39.655836 10304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:39.656173 10304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:39.656230 10304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.656273 10304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:39.656299 10304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:39.662672 10304 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:34929
I20260925 05:10:39.662724 10432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:34929 every 8 connection(s)
I20260925 05:10:39.663133 10304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:39.665310 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 10304
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.189@KRBTEST.COM
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.188@KRBTEST.COM
Sep 25 05:10:39 dist-test-slave-1lvv krb5kdc[9152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313039, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:39.675383 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.664487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:54229 (local address 127.3.6.188:40615)
0925 05:10:39.664555 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.664557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.664630 (+ 73us) server_negotiation.cc:407] Connection header received
0925 05:10:39.665013 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.665015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.665027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.665061 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.666205 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.666983 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.668178 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.668480 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.668543 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.672568 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.672569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.672569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.672578 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.674187 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.674563 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.674564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.674564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.674616 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.674820 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.674821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.674821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.674940 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.674982 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.675188 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.675272 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:10:39.675814 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.664360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.3.6.131:54229)
0925 05:10:39.664574 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.664585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.664913 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.665172 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.665177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.665779 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.666083 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.666089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.667092 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.667092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.668088 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.668091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.668591 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.668591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.668700 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.669454 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.669459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.672437 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.674268 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.674269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.674270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.674464 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.674689 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.674690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.674690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.674746 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.675031 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.675031 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.675079 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.675513 (+ 434us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.675616 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":69,"thread_start_us":74,"threads_started":1}
I20260925 05:10:39.676007 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.664053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:58001 (local address 127.3.6.189:40491)
0925 05:10:39.664099 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.664101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.664584 (+ 483us) server_negotiation.cc:407] Connection header received
0925 05:10:39.665014 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.665015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.665027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.665072 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.666208 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.666722 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.668062 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.668378 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.668444 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.671607 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.671610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.671610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.671618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.673656 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.674181 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.674182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.674182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.674239 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.674519 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.674520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.674520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.674661 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.674723 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.675831 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.675916 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:10:39.676429 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.664045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.3.6.131:58001)
0925 05:10:39.664512 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.664522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.664913 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.665172 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.665177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.665748 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.666083 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.666090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.666837 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.666838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.667968 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.667970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.668539 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.668540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.668701 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.669429 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.669441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.671479 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.673747 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.673755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.673758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.674088 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.674307 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.674307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.674308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.674441 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.674746 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.674754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.674932 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.675062 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.675260 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":13,"spinlock_wait_cycles":384,"thread_start_us":115,"threads_started":1}
I20260925 05:10:39.676663 10433 heartbeater.cc:347] Connected to a master server at 127.3.6.188:40615
I20260925 05:10:39.676986 10435 heartbeater.cc:347] Connected to a master server at 127.3.6.189:40491
I20260925 05:10:39.677057 10435 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.677060 10433 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.677287 10435 heartbeater.cc:511] Master 127.3.6.189:40491 requested a full tablet report, sending...
I20260925 05:10:39.677770 9794 ts_manager.cc:194] Registered new tserver with Master: 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929)
I20260925 05:10:39.679006 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.664054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:40825)
0925 05:10:39.664222 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.664273 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.664886 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.665173 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.665177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.665773 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:39.666083 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.666089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.667173 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.667173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.668893 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.668896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.669911 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.669912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.669981 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.670436 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:39.670441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:39.672738 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:39.677519 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.677520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.677521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.677737 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.678328 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:39.678328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:39.678329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:39.678399 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:39.678695 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:39.678696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:39.678739 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.678792 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.678897 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":39,"thread_start_us":98,"threads_started":1}
I20260925 05:10:39.679368 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.664354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:40825 (local address 127.3.6.190:40397)
0925 05:10:39.664416 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.664418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.664430 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:39.665020 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.665022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.665036 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.665101 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.666205 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.666983 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.669492 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.669783 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.669827 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.673272 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.673274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.673274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.673282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.675404 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.678134 (+ 2730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.678135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.678135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.678210 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.678476 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.678477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.678477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.678605 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.678666 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.679186 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.679266 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:10:39.679952 10434 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40397
I20260925 05:10:39.680008 10434 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:39.680086 10434 heartbeater.cc:511] Master 127.3.6.190:40397 requested a full tablet report, sending...
I20260925 05:10:39.680303 9197 ts_manager.cc:194] Re-registered known tserver with Master: 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929)
I20260925 05:10:39.680739 9197 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:40825
I20260925 05:10:39.677286 10433 heartbeater.cc:511] Master 127.3.6.188:40615 requested a full tablet report, sending...
I20260925 05:10:39.681572 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.671169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.3.6.190:40397)
0925 05:10:39.672016 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.672018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.672490 (+ 472us) server_negotiation.cc:407] Connection header received
0925 05:10:39.672633 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.672634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.672649 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.673390 (+ 741us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.674094 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.674943 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.676109 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.676413 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.676480 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.678003 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.678004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.678005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.678012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.680032 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.680440 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.680441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.680442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.680552 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.680747 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.680748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.680748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.680894 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.680984 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.681335 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.681424 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":360,"threads_started":1}
I20260925 05:10:39.681775 9981 ts_manager.cc:194] Registered new tserver with Master: 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929)
I20260925 05:10:39.681831 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.676203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.0.0.1:41566)
0925 05:10:39.676306 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.676316 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.676348 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.678535 (+ 2187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.678536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.678538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:39.678593 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.678595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.679596 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.679597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.680696 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.680699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.681625 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.681626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.681688 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.681692 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.681722 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.681764 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 05:10:39.682197 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.676495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.3.6.188:40615)
0925 05:10:39.678237 (+ 1742us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.678239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.678282 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:10:39.678307 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.678308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.678334 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.678663 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:39.678702 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.679492 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.680801 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.681509 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.681992 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.682076 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.682094 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.682137 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1721}
I20260925 05:10:39.682402 9991 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 05:10:39.682636 10020 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494ef92f61e24303ad5538bc97340890. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.682694 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.667672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.3.6.189:40491)
0925 05:10:39.667839 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.667841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.667848 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:10:39.667870 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.667870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.667882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.668744 (+ 862us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.668778 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.669503 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.670575 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.670882 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.670990 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.672620 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.672622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.672622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.672629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.677039 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.677506 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.677507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.677507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.677573 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.677802 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.677803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.677803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.678003 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.678097 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.682417 (+ 4320us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.682574 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260925 05:10:39.682739 10020 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: This master's current role is: LEARNER
I20260925 05:10:39.683799 9197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:39.686369 9197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:39.686694 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.672148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.3.6.188:40615)
0925 05:10:39.675491 (+ 3343us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.675493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.678520 (+ 3027us) server_negotiation.cc:407] Connection header received
0925 05:10:39.678543 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.678543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.678553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.678605 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.679588 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.680355 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.681480 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.681748 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.681841 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.683834 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.683835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.683836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.683844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.685386 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.685766 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.685767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.685768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.685832 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.686023 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.686023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.686024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.686164 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.686271 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.686521 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.686613 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3315,"thread_start_us":65,"threads_started":1}
I20260925 05:10:39.687184 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: NON_VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: true } } }
I20260925 05:10:39.687311 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:39.688877 9755 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: attempting to promote NON_VOTER c0d6b213f8fc456a94f2933328964a6a to VOTER
I20260925 05:10:39.689013 9755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:39.689816 9991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 LEARNER]: Refusing update from remote peer 494ef92f61e24303ad5538bc97340890: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 05:10:39.690092 9942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 05:10:39.690174 9804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Refusing update from remote peer 494ef92f61e24303ad5538bc97340890: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 05:10:39.690389 9942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 05:10:39.690861 9756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c0d6b213f8fc456a94f2933328964a6a (127.3.6.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.691035 9991 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c0d6b213f8fc456a94f2933328964a6a (127.3.6.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.691247 9804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c0d6b213f8fc456a94f2933328964a6a (127.3.6.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.691926 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.692008 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:39.692067 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.692131 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:39.692185 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.692245 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890 [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:39.692294 9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.692381 9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:39.692337 10020 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494ef92f61e24303ad5538bc97340890" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:39.692410 10020 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:39.697639 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.693537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36383 (local address 127.0.0.1:60908)
0925 05:10:39.693582 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.693596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.693645 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.694159 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.694161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.694163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:39.694221 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.694223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.695673 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.695674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.696461 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.696464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.697366 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.697367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.697431 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.697436 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.697515 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.697558 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 05:10:39.697660 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.693638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.3.6.129:36383)
0925 05:10:39.693971 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.693977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.693989 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:39.694035 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.694037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.694066 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.694164 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:39.694449 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.695572 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.696556 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.697274 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.697359 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.697438 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.697530 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.697571 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":95,"threads_started":1}
I20260925 05:10:39.698092 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.693861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:34929 (local address 127.0.0.1:53670)
0925 05:10:39.693980 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.693990 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.694034 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.694302 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.694303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.694306 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:39.694352 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.694354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.695466 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.695467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.696712 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.696715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.697872 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.697873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.697933 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.697937 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.697990 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.698031 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":39,"thread_start_us":78,"threads_started":1}
I20260925 05:10:39.698407 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.693959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.3.6.131:34929)
0925 05:10:39.694132 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.694138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.694149 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:39.694191 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.694193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.694211 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.694306 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:39.694414 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.695352 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.696819 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.697666 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.697755 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.697834 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.698275 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.698319 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260925 05:10:39.698457 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.693546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:45535 (local address 127.0.0.1:53612)
0925 05:10:39.693783 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.693795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.693847 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.694349 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.694349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.694351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:39.694384 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.694386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.695830 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.695832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.696897 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.696900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.698234 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.698236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.698299 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.698304 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.698339 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.698380 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":39,"thread_start_us":128,"threads_started":1}
I20260925 05:10:39.698518 10094 tablet_service.cc:1467] Processing CreateTablet for tablet 2189a03fef384d56b9518a69e33d0026 (DEFAULT_TABLE table=test-table [id=b3f2ede38bd244dfa8adf8b55aa64610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:39.698989 10094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2189a03fef384d56b9518a69e33d0026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:39.699142 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.693753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.3.6.130:45535)
0925 05:10:39.694159 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.694164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.694174 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:39.694212 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.694214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.694230 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.694461 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:39.694569 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.695681 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.697012 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.698122 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.698572 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.698970 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.699004 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.699057 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":84,"threads_started":1}
I20260925 05:10:39.699086 10367 tablet_service.cc:1467] Processing CreateTablet for tablet 2189a03fef384d56b9518a69e33d0026 (DEFAULT_TABLE table=test-table [id=b3f2ede38bd244dfa8adf8b55aa64610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:39.699426 10367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2189a03fef384d56b9518a69e33d0026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:39.699895 10231 tablet_service.cc:1467] Processing CreateTablet for tablet 2189a03fef384d56b9518a69e33d0026 (DEFAULT_TABLE table=test-table [id=b3f2ede38bd244dfa8adf8b55aa64610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:39.700280 10231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2189a03fef384d56b9518a69e33d0026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:39.702759 10462 tablet_bootstrap.cc:492] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Bootstrap starting.
I20260925 05:10:39.703231 10463 tablet_bootstrap.cc:492] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Bootstrap starting.
I20260925 05:10:39.703984 10462 tablet_bootstrap.cc:654] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:39.703984 10463 tablet_bootstrap.cc:654] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:39.704289 10464 tablet_bootstrap.cc:492] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Bootstrap starting.
I20260925 05:10:39.704355 10463 log.cc:824] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:39.704487 10462 log.cc:824] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:39.705173 10464 tablet_bootstrap.cc:654] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:39.705207 10463 tablet_bootstrap.cc:492] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: No bootstrap required, opened a new log
I20260925 05:10:39.705269 10463 ts_tablet_manager.cc:1397] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:39.705328 10462 tablet_bootstrap.cc:492] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: No bootstrap required, opened a new log
I20260925 05:10:39.705399 10462 ts_tablet_manager.cc:1397] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 05:10:39.705555 10464 log.cc:824] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:39.706435 10464 tablet_bootstrap.cc:492] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: No bootstrap required, opened a new log
I20260925 05:10:39.706527 10464 ts_tablet_manager.cc:1397] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:39.708086 10462 raft_consensus.cc:348] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.708280 10462 raft_consensus.cc:374] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:39.708315 10462 raft_consensus.cc:729] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563cc985ed2940f0900bddbb1e8be419, State: Initialized, Role: FOLLOWER
I20260925 05:10:39.708467 10462 consensus_queue.cc:260] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.708402 10463 raft_consensus.cc:348] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.708562 10463 raft_consensus.cc:374] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:39.708431 10464 raft_consensus.cc:348] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.708607 10463 raft_consensus.cc:729] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8402cb6315f041fc9846bda9b25af18b, State: Initialized, Role: FOLLOWER
I20260925 05:10:39.708623 10464 raft_consensus.cc:374] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:39.708666 10464 raft_consensus.cc:729] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93740a3f9acf4c9f8b167d7c98ec132d, State: Initialized, Role: FOLLOWER
I20260925 05:10:39.708760 10462 ts_tablet_manager.cc:1428] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260925 05:10:39.708747 10463 consensus_queue.cc:260] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.708796 10464 consensus_queue.cc:260] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.708916 10434 heartbeater.cc:503] Master 127.3.6.190:40397 was elected leader, sending a full tablet report...
I20260925 05:10:39.709038 10463 ts_tablet_manager.cc:1428] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260925 05:10:39.709082 10464 ts_tablet_manager.cc:1428] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:10:39.709344 10298 heartbeater.cc:503] Master 127.3.6.190:40397 was elected leader, sending a full tablet report...
I20260925 05:10:39.710520 10161 heartbeater.cc:503] Master 127.3.6.190:40397 was elected leader, sending a full tablet report...
I20260925 05:10:39.715049 10469 raft_consensus.cc:482] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:39.715191 10469 raft_consensus.cc:504] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.715534 10469 leader_election.cc:288] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535), 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929)
I20260925 05:10:39.718904 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.716127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:48129 (local address 127.3.6.131:34929)
0925 05:10:39.716169 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.716171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.716180 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:39.716204 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.716205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.716220 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.716289 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:39.716450 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.716915 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.718119 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.718524 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.718614 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.718624 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.718792 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.718833 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:10:39.718896 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.715864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56345 (local address 127.3.6.130:45535)
0925 05:10:39.715909 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.715911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.715941 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:10:39.716024 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.716025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.716035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.716084 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:39.716328 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.716823 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.718115 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.718503 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.718595 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.718607 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.718774 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.718818 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:39.718920 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.715834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:45535 (local address 127.3.6.129:56345)
0925 05:10:39.715881 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.715897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.715961 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.716172 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.716172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.716174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:39.716247 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.716248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.716926 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.716928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.718011 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.718014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.718594 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.718595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.718696 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.718699 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.718768 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.718818 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":27}
I20260925 05:10:39.718930 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.716048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:34929 (local address 127.3.6.129:48129)
0925 05:10:39.716080 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:39.716089 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:39.716117 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:39.716294 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:39.716295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:39.716297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:39.716365 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.716366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.717005 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.717006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.718010 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:39.718013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.718616 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.718617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:39.718694 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.718697 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:39.718781 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 05:10:39.718831 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":5}
I20260925 05:10:39.719338 10387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2189a03fef384d56b9518a69e33d0026" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563cc985ed2940f0900bddbb1e8be419" is_pre_election: true
I20260925 05:10:39.719390 10251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2189a03fef384d56b9518a69e33d0026" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" is_pre_election: true
I20260925 05:10:39.719515 10387 raft_consensus.cc:2454] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 0.
I20260925 05:10:39.719798 10048 leader_election.cc:302] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563cc985ed2940f0900bddbb1e8be419, 8402cb6315f041fc9846bda9b25af18b; no voters:
I20260925 05:10:39.719543 10251 raft_consensus.cc:2454] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 0.
I20260925 05:10:39.719947 10469 raft_consensus.cc:2781] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:39.719990 10469 raft_consensus.cc:482] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:39.720027 10469 raft_consensus.cc:3037] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:39.720794 10469 raft_consensus.cc:504] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.720952 10469 leader_election.cc:288] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 election: Requested vote from peers 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535), 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929)
I20260925 05:10:39.721181 10251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2189a03fef384d56b9518a69e33d0026" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93740a3f9acf4c9f8b167d7c98ec132d"
I20260925 05:10:39.721191 10387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2189a03fef384d56b9518a69e33d0026" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563cc985ed2940f0900bddbb1e8be419"
I20260925 05:10:39.721272 10251 raft_consensus.cc:3037] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:39.721308 10387 raft_consensus.cc:3037] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:39.722036 10251 raft_consensus.cc:2454] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 1.
I20260925 05:10:39.722114 10387 raft_consensus.cc:2454] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 1.
I20260925 05:10:39.722210 10048 leader_election.cc:302] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8402cb6315f041fc9846bda9b25af18b, 93740a3f9acf4c9f8b167d7c98ec132d; no voters:
I20260925 05:10:39.722319 10469 raft_consensus.cc:2781] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:39.722496 10469 raft_consensus.cc:686] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 1 LEADER]: Becoming Leader. State: Replica: 8402cb6315f041fc9846bda9b25af18b, State: Running, Role: LEADER
I20260925 05:10:39.722609 10469 consensus_queue.cc:237] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } }
I20260925 05:10:39.723508 9197 catalog_manager.cc:5495] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8402cb6315f041fc9846bda9b25af18b (127.3.6.129). New cstate: current_term: 1 leader_uuid: "8402cb6315f041fc9846bda9b25af18b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } health_report { overall_health: HEALTHY } } }
I20260925 05:10:39.737259 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.734726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.3.6.129:36383)
0925 05:10:39.734802 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.734805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.734814 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:39.734866 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.734867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.734890 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.734974 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:39.735122 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.735607 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.736417 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.736627 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.736704 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.736886 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:39.737008 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:39.737117 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.737135 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.737160 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:10:39.739337 10251 raft_consensus.cc:1261] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Refusing update from remote peer 8402cb6315f041fc9846bda9b25af18b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:39.739440 10387 raft_consensus.cc:1261] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Refusing update from remote peer 8402cb6315f041fc9846bda9b25af18b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:39.739843 10471 consensus_queue.cc:1035] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:39.740039 10469 consensus_queue.cc:1035] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [LEADER]: Connected to new peer: Peer: permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:39.744735 10475 mvcc.cc:205] Tried to move back new op lower bound from 7333122210769289216 to 7333122210704490496. Current Snapshot: MvccSnapshot[applied={T|T < 7333122210769289216 or (T in {7333122210769289216})}]
I20260925 05:10:39.746022 9197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:39.746133 9197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:39.746316 10476 mvcc.cc:205] Tried to move back new op lower bound from 7333122210769289216 to 7333122210704490496. Current Snapshot: MvccSnapshot[applied={T|T < 7333122210769289216 or (T in {7333122210769289216})}]
I20260925 05:10:39.747896 9197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 494ef92f61e24303ad5538bc97340890: Sending DeleteTablet for 3 replicas of tablet 2189a03fef384d56b9518a69e33d0026
I20260925 05:10:39.748245 10231 tablet_service.cc:1514] Processing DeleteTablet for tablet 2189a03fef384d56b9518a69e33d0026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:39 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:53612
I20260925 05:10:39.748274 10094 tablet_service.cc:1514] Processing DeleteTablet for tablet 2189a03fef384d56b9518a69e33d0026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:39 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:60908
I20260925 05:10:39.748274 10367 tablet_service.cc:1514] Processing DeleteTablet for tablet 2189a03fef384d56b9518a69e33d0026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:39 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260925 05:10:39.748514 10486 tablet_replica.cc:330] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: stopping tablet replica
I20260925 05:10:39.748508 10485 tablet_replica.cc:330] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: stopping tablet replica
I20260925 05:10:39.748617 10485 raft_consensus.cc:2229] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:39.748617 10486 raft_consensus.cc:2229] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:39.748723 10486 raft_consensus.cc:2258] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:39.748723 10485 raft_consensus.cc:2258] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:39.749210 10486 ts_tablet_manager.cc:1905] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:39.749286 10485 ts_tablet_manager.cc:1905] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:39.749752 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9165
I20260925 05:10:39.751806 10486 ts_tablet_manager.cc:1918] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:39.751895 10486 log.cc:1196] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/2189a03fef384d56b9518a69e33d0026
I20260925 05:10:39.752206 10486 ts_tablet_manager.cc:1939] T 2189a03fef384d56b9518a69e33d0026 P 563cc985ed2940f0900bddbb1e8be419: Deleting consensus metadata
I20260925 05:10:39.752475 10485 ts_tablet_manager.cc:1918] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:39.752547 10485 log.cc:1196] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/2189a03fef384d56b9518a69e33d0026
I20260925 05:10:39.752817 10485 ts_tablet_manager.cc:1939] T 2189a03fef384d56b9518a69e33d0026 P 8402cb6315f041fc9846bda9b25af18b: Deleting consensus metadata
W20260925 05:10:39.756356 10322 connection.cc:535] server connection from 127.0.0.1:53670 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 05:10:39.756377 10047 connection.cc:535] server connection from 127.0.0.1:60908 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:10:39.756253 10484 tablet_replica.cc:330] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: stopping tablet replica
I20260925 05:10:39.756763 10484 raft_consensus.cc:2229] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:39.756906 10484 raft_consensus.cc:2258] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:39.757485 10484 ts_tablet_manager.cc:1905] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:39.759406 10484 ts_tablet_manager.cc:1918] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:39.759495 10484 log.cc:1196] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/2189a03fef384d56b9518a69e33d0026
I20260925 05:10:39.759852 10484 ts_tablet_manager.cc:1939] T 2189a03fef384d56b9518a69e33d0026 P 93740a3f9acf4c9f8b167d7c98ec132d: Deleting consensus metadata
W20260925 05:10:39.760216 10187 connection.cc:441] server connection from 127.0.0.1:53612 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53612 (request call id 1) could send its response
I20260925 05:10:39.765251 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.749190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.3.6.188:40615)
0925 05:10:39.749241 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.749243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.756009 (+ 6766us) server_negotiation.cc:407] Connection header received
0925 05:10:39.756163 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.756165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.756182 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.756250 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.757776 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.758542 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.759830 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.760136 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.760177 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.762030 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.762031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.762032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.762040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.764053 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.764428 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.764429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.764429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.764485 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.764665 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.764665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.764666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.764800 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.764877 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.765078 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.765153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:10:39.767359 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.758521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.3.6.189:40491)
0925 05:10:39.758576 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.758578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.759109 (+ 531us) server_negotiation.cc:407] Connection header received
0925 05:10:39.759217 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.759218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.759233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.759261 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.760096 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.760823 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.761998 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.762338 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.762414 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.763849 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.763850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.763850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.763858 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.765880 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.766342 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.766344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.766345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.766413 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.766650 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.766651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.766652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.766809 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.766883 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.767075 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.767166 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:10:39.769887 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:39.759014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.3.6.188:40615)
0925 05:10:39.759056 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:39.759058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:39.762210 (+ 3152us) server_negotiation.cc:407] Connection header received
0925 05:10:39.762321 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:39.762322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:39.762335 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:39.762365 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:39.763028 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.763832 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.764935 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:39.765232 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:39.765275 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:39.766776 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:39.766777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:39.766778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:39.766784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:39.768680 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.769106 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.769107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.769107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.769173 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:39.769358 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:39.769359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:39.769359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:39.769513 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:39.769618 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:39.769695 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 05:10:39.769784 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:40.744903 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:40.744571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:47875)
0925 05:10:40.744727 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:40.744815 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
W20260925 05:10:40.745247 10298 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:40397 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:40.745477 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:40.745221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:54611)
0925 05:10:40.745356 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:40.745432 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
W20260925 05:10:40.745645 10434 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:40397 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:40.745889 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:40.745532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:55969)
0925 05:10:40.745774 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:40.745855 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":53,"threads_started":1}
W20260925 05:10:40.746114 10161 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:40397 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:41.248333 10498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 494ef92f61e24303ad5538bc97340890)
I20260925 05:10:41.248498 10499 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 494ef92f61e24303ad5538bc97340890)
I20260925 05:10:41.248526 10498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:41.248646 10499 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:41.248871 10498 leader_election.cc:288] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397), c0d6b213f8fc456a94f2933328964a6a (127.3.6.188:40615)
I20260925 05:10:41.249166 10499 leader_election.cc:288] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397), 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189:40491)
I20260925 05:10:41.249282 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.249039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41106)
0925 05:10:41.249176 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.249241 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260925 05:10:41.249435 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.249046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41112)
0925 05:10:41.249192 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.249311 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":53,"threads_started":1}
W20260925 05:10:41.249439 9778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
W20260925 05:10:41.249650 9965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:41.249969 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.249878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41120)
0925 05:10:41.249914 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.249944 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 05:10:41.250074 9778 leader_election.cc:334] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397): Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:41.250259 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.250147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41124)
0925 05:10:41.250199 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.250227 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 05:10:41.250336 9965 leader_election.cc:334] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397): Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:41.253264 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.249406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.3.6.188:40615)
0925 05:10:41.249486 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:41.249488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:41.249498 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:41.249864 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:41.249866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:41.249875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:41.249944 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:41.250113 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.250955 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.252489 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.252860 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.252957 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.252969 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:41.253121 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 05:10:41.253161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":40,"threads_started":1}
I20260925 05:10:41.253264 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.249213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:40615 (local address 127.0.0.1:41596)
0925 05:10:41.249312 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.249325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:41.249667 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:41.249946 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:41.249947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:41.249949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:41.250017 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.250019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.251053 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.251054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.252385 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.252389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.252958 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.252959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.253032 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.253035 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:41.253113 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:10:41.253161 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260925 05:10:41.253268 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.249654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.3.6.189:40491)
0925 05:10:41.249793 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:41.249795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:41.249803 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:10:41.249831 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:41.249832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:41.249842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:41.250179 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:41.250325 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.251051 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.252514 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.252902 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.252989 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.253000 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:41.253147 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:10:41.253189 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260925 05:10:41.253268 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.249514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:40491 (local address 127.0.0.1:43648)
0925 05:10:41.249582 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.249593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:41.249654 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:41.250192 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:41.250193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:41.250195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:41.250240 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.250242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.251123 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.251124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.252386 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:41.252389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:41.252989 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:41.252990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:41.253058 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:41.253061 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:41.253138 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:10:41.253186 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"thread_start_us":54,"threads_started":1}
I20260925 05:10:41.253527 9991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0d6b213f8fc456a94f2933328964a6a" is_pre_election: true
I20260925 05:10:41.253749 9991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ef226760b44a5a899e1faf6b9b0fc8 in term 1.
I20260925 05:10:41.253925 9778 leader_election.cc:302] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92ef226760b44a5a899e1faf6b9b0fc8, c0d6b213f8fc456a94f2933328964a6a; no voters: 494ef92f61e24303ad5538bc97340890
I20260925 05:10:41.253922 9804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d6b213f8fc456a94f2933328964a6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" is_pre_election: true
I20260925 05:10:41.254071 9804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d6b213f8fc456a94f2933328964a6a in term 1.
I20260925 05:10:41.254113 10498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 05:10:41.254166 10498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 494ef92f61e24303ad5538bc97340890)
I20260925 05:10:41.254196 10498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:10:41.254248 9967 leader_election.cc:302] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92ef226760b44a5a899e1faf6b9b0fc8, c0d6b213f8fc456a94f2933328964a6a; no voters: 494ef92f61e24303ad5538bc97340890
I20260925 05:10:41.254361 10499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 05:10:41.254426 10499 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 494ef92f61e24303ad5538bc97340890)
I20260925 05:10:41.254456 10499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:10:41.255050 10498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:41.255146 10499 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:41.255208 10498 leader_election.cc:288] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [CANDIDATE]: Term 2 election: Requested vote from peers 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397), c0d6b213f8fc456a94f2933328964a6a (127.3.6.188:40615)
I20260925 05:10:41.255294 10499 leader_election.cc:288] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 2 election: Requested vote from peers 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397), 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189:40491)
I20260925 05:10:41.255450 9991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0d6b213f8fc456a94f2933328964a6a"
I20260925 05:10:41.255501 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.255371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41154)
0925 05:10:41.255445 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.255476 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:10:41.255540 9991 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 92ef226760b44a5a899e1faf6b9b0fc8 in current term 2: Already voted for candidate c0d6b213f8fc456a94f2933328964a6a in this term.
I20260925 05:10:41.255555 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.255302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41138)
0925 05:10:41.255469 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.255501 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 05:10:41.255801 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.255703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41162)
0925 05:10:41.255734 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.255780 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 05:10:41.255849 9965 leader_election.cc:334] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397): Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:41.255733 9804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d6b213f8fc456a94f2933328964a6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
I20260925 05:10:41.256019 9804 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c0d6b213f8fc456a94f2933328964a6a in current term 2: Already voted for candidate 92ef226760b44a5a899e1faf6b9b0fc8 in this term.
I20260925 05:10:41.256191 9967 leader_election.cc:302] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0d6b213f8fc456a94f2933328964a6a; no voters: 494ef92f61e24303ad5538bc97340890, 92ef226760b44a5a899e1faf6b9b0fc8
I20260925 05:10:41.256232 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.256138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41182)
0925 05:10:41.256184 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.256207 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 05:10:41.256258 10499 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 05:10:41.256286 9778 leader_election.cc:334] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397): Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:41.256320 9778 leader_election.cc:302] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ef226760b44a5a899e1faf6b9b0fc8; no voters: 494ef92f61e24303ad5538bc97340890, c0d6b213f8fc456a94f2933328964a6a
I20260925 05:10:41.256376 10498 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 05:10:41.746124 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.745781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:42485)
0925 05:10:41.746006 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.746055 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260925 05:10:41.746510 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.746079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:57049)
0925 05:10:41.746406 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.746446 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":72,"threads_started":1}
I20260925 05:10:41.746755 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:41.746461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:46613)
0925 05:10:41.746641 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:41.746706 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260925 05:10:42.747161 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.746908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:38273)
0925 05:10:42.747052 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.747098 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260925 05:10:42.747466 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.747017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:58967)
0925 05:10:42.747382 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.747419 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":63,"threads_started":1}
I20260925 05:10:42.747819 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.747244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:46379)
0925 05:10:42.747720 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.747778 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":60,"threads_started":1}
I20260925 05:10:42.945530 10514 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:42.945648 10514 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:42.945955 10514 leader_election.cc:288] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397), 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189:40491)
I20260925 05:10:42.946103 9804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d6b213f8fc456a94f2933328964a6a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" is_pre_election: true
I20260925 05:10:42.946228 9804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d6b213f8fc456a94f2933328964a6a in term 2.
I20260925 05:10:42.946199 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.945958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41436)
0925 05:10:42.946113 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.946165 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260925 05:10:42.946409 9967 leader_election.cc:302] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ef226760b44a5a899e1faf6b9b0fc8, c0d6b213f8fc456a94f2933328964a6a; no voters:
I20260925 05:10:42.946557 10514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 05:10:42.946640 10514 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:42.946669 10514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:10:42.946698 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.946578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41452)
0925 05:10:42.946625 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.946654 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:10:42.946774 9965 leader_election.cc:334] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397): Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:42.947458 10514 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:42.947625 10514 leader_election.cc:288] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 3 election: Requested vote from peers 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397), 92ef226760b44a5a899e1faf6b9b0fc8 (127.3.6.189:40491)
I20260925 05:10:42.947840 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.947703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41460)
0925 05:10:42.947771 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.947815 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 05:10:42.947896 9804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d6b213f8fc456a94f2933328964a6a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92ef226760b44a5a899e1faf6b9b0fc8"
I20260925 05:10:42.947981 9804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:10:42.948199 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:42.948091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41470)
0925 05:10:42.948144 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:42.948175 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 05:10:42.948282 9965 leader_election.cc:334] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397): Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
I20260925 05:10:42.948741 9804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0d6b213f8fc456a94f2933328964a6a in term 3.
I20260925 05:10:42.948920 9967 leader_election.cc:302] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92ef226760b44a5a899e1faf6b9b0fc8, c0d6b213f8fc456a94f2933328964a6a; no voters: 494ef92f61e24303ad5538bc97340890
I20260925 05:10:42.949025 10514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:10:42.949164 10514 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [term 3 LEADER]: Becoming Leader. State: Replica: c0d6b213f8fc456a94f2933328964a6a, State: Running, Role: LEADER
I20260925 05:10:42.949254 10514 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } }
I20260925 05:10:42.949734 10517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d6b213f8fc456a94f2933328964a6a. Latest consensus state: current_term: 3 leader_uuid: "c0d6b213f8fc456a94f2933328964a6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:42.949843 10517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:42.950027 10519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:42.950402 10519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:42.950457 10519 catalog_manager.cc:1259] Loaded cluster ID: 01264a14d0714e6b96ec9098c4721ede
I20260925 05:10:42.950474 10519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:42.950676 10519 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:42.950773 10519 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Loaded TSK: 0
I20260925 05:10:42.950953 10519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:10:43.025215 9981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:43.026835 9981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:43.027920 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.027794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41486)
0925 05:10:43.027857 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.027890 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:10:43.027956 9804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [term 3 FOLLOWER]: Refusing update from remote peer c0d6b213f8fc456a94f2933328964a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 05:10:43.028308 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.028196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.0.0.1:41498)
0925 05:10:43.028253 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.028282 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 05:10:43.028415 10514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 05:10:43.028558 9965 consensus_peers.cc:487] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a -> Peer 494ef92f61e24303ad5538bc97340890 (127.3.6.190:40397): Couldn't send request to peer 494ef92f61e24303ad5538bc97340890. Status: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:10:43.029042 10521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d6b213f8fc456a94f2933328964a6a. Latest consensus state: current_term: 3 leader_uuid: "c0d6b213f8fc456a94f2933328964a6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:43.029176 10521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:43.029631 10514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c0d6b213f8fc456a94f2933328964a6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:43.029723 10514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:43.029734 10521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c0d6b213f8fc456a94f2933328964a6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "494ef92f61e24303ad5538bc97340890" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40397 } } peers { permanent_uuid: "92ef226760b44a5a899e1faf6b9b0fc8" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 40491 } attrs { promote: false } } peers { permanent_uuid: "c0d6b213f8fc456a94f2933328964a6a" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 40615 } attrs { promote: false } } }
I20260925 05:10:43.029832 10521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ef226760b44a5a899e1faf6b9b0fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:10:43.035806 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.031918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.3.6.131:34929)
0925 05:10:43.032042 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:43.032045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:43.032630 (+ 585us) server_negotiation.cc:407] Connection header received
0925 05:10:43.032704 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:43.032705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:43.032718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:43.032794 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:43.032996 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.033675 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.034728 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.035319 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.035403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:43.035420 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:43.035658 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 05:10:43.035709 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260925 05:10:43.035841 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.031861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36383 (local address 127.0.0.1:60936)
0925 05:10:43.032034 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.032045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:43.032104 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:43.032318 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:43.032320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:43.032322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:43.032379 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:43.032382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.033352 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.033353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:43.034668 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:43.034671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.035567 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.035568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:43.035631 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:43.035635 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:43.035691 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:10:43.035730 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":46}
I20260925 05:10:43.035853 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.032206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:34929 (local address 127.0.0.1:53672)
0925 05:10:43.032563 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.032574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:43.032637 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:43.032847 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:43.032848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:43.032851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:43.032895 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:43.032897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.033785 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.033786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:43.034625 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:43.034628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.035472 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.035473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:43.035546 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:43.035552 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:10:43.035642 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:10:43.035697 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260925 05:10:43.036082 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.031914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.3.6.129:36383)
0925 05:10:43.032053 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:43.032055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:43.032093 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:10:43.032195 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:43.032196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:43.032209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:43.032306 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:43.032492 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.033246 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.034992 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.035493 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.035566 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:43.035581 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:43.035971 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 05:10:43.036026 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260925 05:10:43.036528 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.032390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:45535 (local address 127.0.0.1:53626)
0925 05:10:43.032710 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.032716 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:43.032771 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:43.032933 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:43.032934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:43.032936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:43.032984 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:43.032986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.033594 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.033595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:43.034742 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:43.034743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.035473 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.035474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:43.035561 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:43.035566 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:43.035616 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:10:43.035661 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260925 05:10:43.036566 10094 tablet_service.cc:1467] Processing CreateTablet for tablet d00ddc5cfb684dca91aa0807fb251418 (DEFAULT_TABLE table=test-table [id=4674e366d70346e78986c38da0a815b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:43.036621 10367 tablet_service.cc:1467] Processing CreateTablet for tablet d00ddc5cfb684dca91aa0807fb251418 (DEFAULT_TABLE table=test-table [id=4674e366d70346e78986c38da0a815b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:43.036700 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.032469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.3.6.130:45535)
0925 05:10:43.032620 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:43.032622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:43.032763 (+ 141us) server_negotiation.cc:407] Connection header received
0925 05:10:43.032824 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:43.032825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:43.032834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:43.032929 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:43.033037 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.033512 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.034813 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.035388 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.035465 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:43.035478 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:43.035736 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 05:10:43.035812 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260925 05:10:43.036724 10094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00ddc5cfb684dca91aa0807fb251418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:43.036744 10367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00ddc5cfb684dca91aa0807fb251418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:43.036880 10231 tablet_service.cc:1467] Processing CreateTablet for tablet d00ddc5cfb684dca91aa0807fb251418 (DEFAULT_TABLE table=test-table [id=4674e366d70346e78986c38da0a815b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:43.036996 10231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00ddc5cfb684dca91aa0807fb251418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:43.038306 10531 tablet_bootstrap.cc:492] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: Bootstrap starting.
I20260925 05:10:43.038614 10532 tablet_bootstrap.cc:492] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Bootstrap starting.
I20260925 05:10:43.038784 10531 tablet_bootstrap.cc:654] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:43.039016 10532 tablet_bootstrap.cc:654] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:43.039359 10533 tablet_bootstrap.cc:492] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b: Bootstrap starting.
I20260925 05:10:43.039454 10531 tablet_bootstrap.cc:492] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: No bootstrap required, opened a new log
I20260925 05:10:43.039505 10531 ts_tablet_manager.cc:1397] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:43.039638 10531 raft_consensus.cc:348] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.039733 10531 raft_consensus.cc:374] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:43.039749 10533 tablet_bootstrap.cc:654] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:43.039805 10531 raft_consensus.cc:729] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93740a3f9acf4c9f8b167d7c98ec132d, State: Initialized, Role: FOLLOWER
I20260925 05:10:43.039877 10531 consensus_queue.cc:260] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.040236 10531 ts_tablet_manager.cc:1428] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:43.040637 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.040518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:58533)
0925 05:10:43.040573 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.040609 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 05:10:43.040807 10297 heartbeater.cc:503] Master 127.3.6.188:40615 was elected leader, sending a full tablet report...
I20260925 05:10:43.041056 10532 tablet_bootstrap.cc:492] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: No bootstrap required, opened a new log
I20260925 05:10:43.041132 10533 tablet_bootstrap.cc:492] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b: No bootstrap required, opened a new log
I20260925 05:10:43.041155 10532 ts_tablet_manager.cc:1397] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 05:10:43.041201 10533 ts_tablet_manager.cc:1397] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:43.041379 10532 raft_consensus.cc:348] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.041455 10532 raft_consensus.cc:374] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:43.041476 10532 raft_consensus.cc:729] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563cc985ed2940f0900bddbb1e8be419, State: Initialized, Role: FOLLOWER
I20260925 05:10:43.041473 10533 raft_consensus.cc:348] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.041522 10532 consensus_queue.cc:260] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.041564 10533 raft_consensus.cc:374] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:43.041594 10533 raft_consensus.cc:729] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8402cb6315f041fc9846bda9b25af18b, State: Initialized, Role: FOLLOWER
I20260925 05:10:43.041643 10533 consensus_queue.cc:260] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.041725 10532 ts_tablet_manager.cc:1428] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:43.041822 10533 ts_tablet_manager.cc:1428] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:43.042128 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.041991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:39553)
0925 05:10:43.042068 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.042101 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:10:43.042377 10433 heartbeater.cc:503] Master 127.3.6.188:40615 was elected leader, sending a full tablet report...
I20260925 05:10:43.042443 10160 heartbeater.cc:503] Master 127.3.6.188:40615 was elected leader, sending a full tablet report...
I20260925 05:10:43.042546 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.042298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:34147)
0925 05:10:43.042339 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.042367 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 05:10:43.042936 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.042801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:51269)
0925 05:10:43.042846 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.042908 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 05:10:43.042981 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.042827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:58815)
0925 05:10:43.042928 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.042955 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 05:10:43.124823 10163 tablet.cc:2369] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:10:43.166214 10436 tablet.cc:2369] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:43.237344 10539 raft_consensus.cc:482] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:43.237450 10539 raft_consensus.cc:504] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.237768 10539 leader_election.cc:288] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929), 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535)
I20260925 05:10:43.238027 10387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00ddc5cfb684dca91aa0807fb251418" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563cc985ed2940f0900bddbb1e8be419" is_pre_election: true
I20260925 05:10:43.238070 10251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00ddc5cfb684dca91aa0807fb251418" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" is_pre_election: true
I20260925 05:10:43.238152 10387 raft_consensus.cc:2454] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 0.
I20260925 05:10:43.238201 10251 raft_consensus.cc:2454] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 0.
I20260925 05:10:43.238351 10048 leader_election.cc:302] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8402cb6315f041fc9846bda9b25af18b, 93740a3f9acf4c9f8b167d7c98ec132d; no voters:
I20260925 05:10:43.238466 10539 raft_consensus.cc:2781] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:43.238499 10539 raft_consensus.cc:482] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:43.238523 10539 raft_consensus.cc:3037] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:43.239318 10539 raft_consensus.cc:504] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.239485 10539 leader_election.cc:288] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 election: Requested vote from peers 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929), 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535)
I20260925 05:10:43.239651 10387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00ddc5cfb684dca91aa0807fb251418" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563cc985ed2940f0900bddbb1e8be419"
I20260925 05:10:43.239674 10251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00ddc5cfb684dca91aa0807fb251418" candidate_uuid: "8402cb6315f041fc9846bda9b25af18b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93740a3f9acf4c9f8b167d7c98ec132d"
I20260925 05:10:43.239750 10251 raft_consensus.cc:3037] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:43.239750 10387 raft_consensus.cc:3037] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:43.240372 10251 raft_consensus.cc:2454] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 1.
I20260925 05:10:43.240540 10387 raft_consensus.cc:2454] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8402cb6315f041fc9846bda9b25af18b in term 1.
I20260925 05:10:43.240579 10048 leader_election.cc:302] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8402cb6315f041fc9846bda9b25af18b, 93740a3f9acf4c9f8b167d7c98ec132d; no voters:
I20260925 05:10:43.240722 10539 raft_consensus.cc:2781] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:43.240909 10539 raft_consensus.cc:686] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 1 LEADER]: Becoming Leader. State: Replica: 8402cb6315f041fc9846bda9b25af18b, State: Running, Role: LEADER
I20260925 05:10:43.240972 10539 consensus_queue.cc:237] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } }
I20260925 05:10:43.241293 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.241169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:60097)
0925 05:10:43.241212 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.241255 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 05:10:43.241632 9980 catalog_manager.cc:5495] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8402cb6315f041fc9846bda9b25af18b (127.3.6.129). New cstate: current_term: 1 leader_uuid: "8402cb6315f041fc9846bda9b25af18b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8402cb6315f041fc9846bda9b25af18b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:43.263006 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.260845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.3.6.129:36383)
0925 05:10:43.260904 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:43.260906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:43.260915 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:43.260964 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:43.260965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:43.260977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:43.261025 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:43.261144 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.261617 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.262366 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:43.262559 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:43.262626 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:43.262746 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:43.262773 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:43.262812 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:43.262869 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 05:10:43.262889 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:10:43.264138 10251 raft_consensus.cc:1261] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Refusing update from remote peer 8402cb6315f041fc9846bda9b25af18b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:43.264143 10387 raft_consensus.cc:1261] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Refusing update from remote peer 8402cb6315f041fc9846bda9b25af18b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:43.264420 10540 consensus_queue.cc:1035] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [LEADER]: Connected to new peer: Peer: permanent_uuid: "563cc985ed2940f0900bddbb1e8be419" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 34929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:43.264554 10539 consensus_queue.cc:1035] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93740a3f9acf4c9f8b167d7c98ec132d" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:43.265594 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.265450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:35503)
0925 05:10:43.265516 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.265557 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:10:43.265635 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.265338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:57843)
0925 05:10:43.265568 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.265603 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260925 05:10:43.266335 10543 mvcc.cc:205] Tried to move back new op lower bound from 7333122225207681024 to 7333122225115545600. Current Snapshot: MvccSnapshot[applied={T|T < 7333122225207681024}]
I20260925 05:10:43.266484 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.266308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:37859)
0925 05:10:43.266416 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.266447 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 05:10:43.266769 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.266685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.129:38503)
0925 05:10:43.266715 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.266744 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 05:10:43.267539 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.267387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.130:39363)
0925 05:10:43.267462 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.267508 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 05:10:43.268107 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.267987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40397 (local address 127.3.6.131:57965)
0925 05:10:43.268050 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.268076 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:40397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 05:10:43.268085 9981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:43.268173 9981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:43.269519 9981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0d6b213f8fc456a94f2933328964a6a: Sending DeleteTablet for 3 replicas of tablet d00ddc5cfb684dca91aa0807fb251418
I20260925 05:10:43.269747 10231 tablet_service.cc:1514] Processing DeleteTablet for tablet d00ddc5cfb684dca91aa0807fb251418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:43 UTC) from {username='oryx', principal='oryx/127.3.6.188@KRBTEST.COM'} at 127.0.0.1:53626
I20260925 05:10:43.269860 10367 tablet_service.cc:1514] Processing DeleteTablet for tablet d00ddc5cfb684dca91aa0807fb251418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:43 UTC) from {username='oryx', principal='oryx/127.3.6.188@KRBTEST.COM'} at 127.0.0.1:53672
I20260925 05:10:43.270069 10552 tablet_replica.cc:330] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: stopping tablet replica
I20260925 05:10:43.270153 10552 raft_consensus.cc:2229] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:43.270159 10094 tablet_service.cc:1514] Processing DeleteTablet for tablet d00ddc5cfb684dca91aa0807fb251418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:43 UTC) from {username='oryx', principal='oryx/127.3.6.188@KRBTEST.COM'} at 127.0.0.1:60936
I20260925 05:10:43.270206 10552 raft_consensus.cc:2258] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:43.270363 10553 tablet_replica.cc:330] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b: stopping tablet replica
I20260925 05:10:43.270401 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9941
I20260925 05:10:43.270424 10551 tablet_replica.cc:330] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: stopping tablet replica
I20260925 05:10:43.270458 10553 raft_consensus.cc:2229] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:43.270668 10553 raft_consensus.cc:2258] T d00ddc5cfb684dca91aa0807fb251418 P 8402cb6315f041fc9846bda9b25af18b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:43.271445 10552 ts_tablet_manager.cc:1905] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:43.270484 10551 raft_consensus.cc:2229] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:43.270619 10231 tablet_service.cc:1514] Processing DeleteTablet for tablet d00ddc5cfb684dca91aa0807fb251418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:43 UTC) from {username='oryx', principal='oryx/127.3.6.188@KRBTEST.COM'} at 127.0.0.1:53626
I20260925 05:10:43.272653 10551 raft_consensus.cc:2258] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:43.272953 10551 ts_tablet_manager.cc:1905] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:43.273434 10552 ts_tablet_manager.cc:1918] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:43.273496 10552 log.cc:1196] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/d00ddc5cfb684dca91aa0807fb251418
I20260925 05:10:43.273728 10552 ts_tablet_manager.cc:1939] T d00ddc5cfb684dca91aa0807fb251418 P 563cc985ed2940f0900bddbb1e8be419: Deleting consensus metadata
I20260925 05:10:43.274345 9966 catalog_manager.cc:4841] TS 563cc985ed2940f0900bddbb1e8be419 (127.3.6.131:34929): tablet d00ddc5cfb684dca91aa0807fb251418 (table test-table [id=4674e366d70346e78986c38da0a815b6]) successfully deleted
I20260925 05:10:43.274510 10551 ts_tablet_manager.cc:1918] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:43.274567 10551 log.cc:1196] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/d00ddc5cfb684dca91aa0807fb251418
I20260925 05:10:43.274787 10551 ts_tablet_manager.cc:1939] T d00ddc5cfb684dca91aa0807fb251418 P 93740a3f9acf4c9f8b167d7c98ec132d: Deleting consensus metadata
I20260925 05:10:43.275084 9966 catalog_manager.cc:4841] TS 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535): tablet d00ddc5cfb684dca91aa0807fb251418 (table test-table [id=4674e366d70346e78986c38da0a815b6]) successfully deleted
W20260925 05:10:43.275210 9966 catalog_manager.cc:4808] TS 93740a3f9acf4c9f8b167d7c98ec132d (127.3.6.130:45535): delete failed for tablet d00ddc5cfb684dca91aa0807fb251418 because the tablet was not found. No further retry: Not found: Tablet not found: d00ddc5cfb684dca91aa0807fb251418
I20260925 05:10:43.278555 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:43.278416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36383 (local address 127.0.0.1:60966)
0925 05:10:43.278479 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:43.278511 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 05:10:43.278744 9968 catalog_manager.cc:4584] TS 8402cb6315f041fc9846bda9b25af18b (127.3.6.129:36383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d00ddc5cfb684dca91aa0807fb251418: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36383: connect: Connection refused (error 111)
I20260925 05:10:43.278931 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 10168
I20260925 05:10:43.285148 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 10304
I20260925 05:10:43.290333 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9661
I20260925 05:10:43.295552 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 9848
2026-09-25T05:10:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6442 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:43 dist-test-slave-1lvv krb5kdc[10556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:43 dist-test-slave-1lvv krb5kdc[10556](info): set up 2 sockets
Sep 25 05:10:43 dist-test-slave-1lvv krb5kdc[10556](info): commencing operation
krb5kdc: starting...
W20260925 05:10:44.348634 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:44 dist-test-slave-1lvv krb5kdc[10556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab.
Entry for principal oryx/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab.
I20260925 05:10:44.482911 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45435
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab
--principal=oryx/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:44.567971 10569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:44.568217 10569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:44.568243 10569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:44.569501 10569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:44.569548 10569 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:10:44.569563 10569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:44.569610 10569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:44.569638 10569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:44.569655 10569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:44.571228 10569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:45435
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.190.keytab
--principal=oryx/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:44.571499 10569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:44.571795 10569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:44.577298 10574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:44.577423 10575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:44.577301 10577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:44.577513 10569 server_base.cc:1034] running on GCE node
I20260925 05:10:44.577867 10569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:44.578181 10569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:44.579351 10569 hybrid_clock.cc:648] HybridClock initialized: now 1790313044579333 us; error 36 us; skew 500 ppm
Sep 25 05:10:44 dist-test-slave-1lvv krb5kdc[10556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:44.582041 10569 init.cc:377] Logged in from keytab as oryx/127.3.6.190@KRBTEST.COM (short username oryx)
I20260925 05:10:44.582481 10569 webserver.cc:460] Webserver started at http://127.3.6.190:38765/ using document root <none> and password file <none>
I20260925 05:10:44.582679 10569 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:44.582722 10569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:44.582821 10569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:44.584046 10569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "487476619baf54e936a8acf93b0bd943"
server_key_iv: "674cbfbd097b8127ac278ef115d30fdc"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.584394 10569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "487476619baf54e936a8acf93b0bd943"
server_key_iv: "674cbfbd097b8127ac278ef115d30fdc"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.585807 10569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:44.586555 10584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:44.586767 10569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:44.586880 10569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "487476619baf54e936a8acf93b0bd943"
server_key_iv: "674cbfbd097b8127ac278ef115d30fdc"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.586977 10569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:44.605414 10569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:44.608108 10569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:44.608297 10569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:44.613317 10569 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:45435
I20260925 05:10:44.613405 10646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:45435 every 8 connection(s)
I20260925 05:10:44.613796 10569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:44.614580 10647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:44.617079 10647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Bootstrap starting.
I20260925 05:10:44.617825 10647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:44.618170 10647 log.cc:824] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:44.618953 10647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: No bootstrap required, opened a new log
I20260925 05:10:44.619032 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 10569
I20260925 05:10:44.619143 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:44.619364 3098 external_mini_cluster.cc:1428] Setting key 625e5c4bb1857ec31c8286d31121f369
I20260925 05:10:44.621948 10647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45435 } }
I20260925 05:10:44.622160 10647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:44.622205 10647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1af451a7b34f77b9ace9beb1dcd0ec, State: Initialized, Role: FOLLOWER
I20260925 05:10:44.622366 10647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45435 } }
I20260925 05:10:44.622459 10647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:44.622484 10647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:44.622552 10647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:44.623361 10647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45435 } }
I20260925 05:10:44.623474 10647 leader_election.cc:302] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f1af451a7b34f77b9ace9beb1dcd0ec; no voters:
I20260925 05:10:44.623657 10647 leader_election.cc:288] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:44.623746 10652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:44.623939 10647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:44.623973 10652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [term 1 LEADER]: Becoming Leader. State: Replica: 8f1af451a7b34f77b9ace9beb1dcd0ec, State: Running, Role: LEADER
I20260925 05:10:44.624080 10652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45435 } }
I20260925 05:10:44.624410 10652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f1af451a7b34f77b9ace9beb1dcd0ec. Latest consensus state: current_term: 1 leader_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45435 } } }
I20260925 05:10:44.624509 10652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:44.624420 10653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1af451a7b34f77b9ace9beb1dcd0ec" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45435 } } }
I20260925 05:10:44.624663 10653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:44.624775 10661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:44.625357 10661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:44.627187 10661 catalog_manager.cc:1347] Generated new cluster ID: b4fd2e343c7b43ca8510a289018c9449
I20260925 05:10:44.627254 10661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:10:44 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:44.632441 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.620009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.3.6.190:45435)
0925 05:10:44.620221 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:44.620232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:44.620269 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:10:44.621888 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:44.621898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:44.621970 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:44.622137 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:44.623030 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.624502 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.625715 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.626139 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.626268 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.628918 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:44.628922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:44.628922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:44.628934 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:44.630823 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.631197 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.631200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.631201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.631292 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.631514 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.631515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.631515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.631825 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:44.632010 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:44.632122 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 05:10:44.632270 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260925 05:10:44.635278 10661 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:44.635898 10661 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:44.638473 10661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Generated new TSK 0
I20260925 05:10:44.638696 10661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab.
Entry for principal oryx/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab.
I20260925 05:10:44.672026 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45435
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab
--principal=oryx/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:44.758148 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:44.758374 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:44.758401 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:44.759603 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:44.759649 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:44.759670 10673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:44.759688 10673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:44.761309 10673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab
--principal=oryx/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:44.761600 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:44.761849 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:44.767215 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:44.767338 10673 server_base.cc:1034] running on GCE node
W20260925 05:10:44.767208 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:44.767434 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:44.767714 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:44.768042 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:44.769243 10673 hybrid_clock.cc:648] HybridClock initialized: now 1790313044769236 us; error 41 us; skew 500 ppm
Sep 25 05:10:44 dist-test-slave-1lvv krb5kdc[10556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:44.771981 10673 init.cc:377] Logged in from keytab as oryx/127.3.6.129@KRBTEST.COM (short username oryx)
I20260925 05:10:44.772478 10673 webserver.cc:460] Webserver started at http://127.3.6.129:38207/ using document root <none> and password file <none>
I20260925 05:10:44.772742 10673 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:44.772815 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:44.772974 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:44.774101 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "08251cc1c3ce4aa29a946868c131a007"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "2e5c1b0c5b43e5150d1bd7c24947a1bb"
server_key_iv: "3f00b2ce19f8875a1e6595c04e9319d2"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.774505 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "08251cc1c3ce4aa29a946868c131a007"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "2e5c1b0c5b43e5150d1bd7c24947a1bb"
server_key_iv: "3f00b2ce19f8875a1e6595c04e9319d2"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.776046 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:44.776850 10688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:44.777032 10673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:44.777143 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "08251cc1c3ce4aa29a946868c131a007"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "2e5c1b0c5b43e5150d1bd7c24947a1bb"
server_key_iv: "3f00b2ce19f8875a1e6595c04e9319d2"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.777230 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:44.805357 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:44.807585 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:44.807721 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:44.808513 10673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:44.808583 10673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:44.808619 10673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:44.808643 10673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:44.815379 10673 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:40477
I20260925 05:10:44.815474 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:40477 every 8 connection(s)
I20260925 05:10:44.815846 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:10:44 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:44.818128 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 10673
I20260925 05:10:44.818262 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:44.818514 3098 external_mini_cluster.cc:1428] Setting key 047631267169cf3f2731fde8636d8b91
I20260925 05:10:44.820004 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.808799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:39307 (local address 127.3.6.190:45435)
0925 05:10:44.808850 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:44.808853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:44.809088 (+ 235us) server_negotiation.cc:407] Connection header received
0925 05:10:44.809664 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:44.809666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:44.809677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:44.809737 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:44.810706 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.811591 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.812804 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.813033 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.813126 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.816465 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:44.816466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:44.816467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:44.816473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:44.818174 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.818769 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.818770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.818770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.818866 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.819214 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.819215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.819216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.819373 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:44.819451 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:44.819720 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:10:44.819863 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:10:44.820046 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.808807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45435 (local address 127.3.6.129:39307)
0925 05:10:44.808951 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:44.808987 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:44.809526 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:44.809837 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:44.809843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:44.810249 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:44.810556 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.810575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.811715 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.811716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.812700 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.812702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.813159 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.813160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.813323 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.814464 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:44.814475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:44.816370 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:44.818287 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.818297 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.818299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.818648 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.818952 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.818953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.818953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.819096 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.819451 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:44.819457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:44.819582 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 05:10:44.819676 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:10:44.819876 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260925 05:10:44.820574 10804 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45435
I20260925 05:10:44.820655 10804 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:44.820849 10804 heartbeater.cc:511] Master 127.3.6.190:45435 requested a full tablet report, sending...
I20260925 05:10:44.821388 10601 ts_manager.cc:194] Registered new tserver with Master: 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477)
I20260925 05:10:44.822311 10601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:39307
I20260925 05:10:44.827929 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.821652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:37847 (local address 127.3.6.190:45435)
0925 05:10:44.821692 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:44.821694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:44.821974 (+ 280us) server_negotiation.cc:407] Connection header received
0925 05:10:44.822007 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:44.822008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:44.822020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:44.822083 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:44.822420 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.822956 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.824025 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.824214 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.824272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.825361 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:44.825362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:44.825362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:44.825368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:44.826700 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.827119 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.827119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.827119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.827177 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.827397 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.827397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.827398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.827492 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:44.827541 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:44.827716 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 05:10:44.827825 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:44.827948 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.821725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45435 (local address 127.3.6.129:37847)
0925 05:10:44.821889 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:44.821901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:44.821965 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:44.822086 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:44.822086 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:44.822261 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:44.822336 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.822338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.823056 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.823057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.823949 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.823952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.824294 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.824295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.824364 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.824793 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:44.824798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:44.825259 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:44.826787 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.826788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.826789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.827035 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.827254 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.827255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.827256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.827330 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.827556 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:44.827557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:44.827616 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 05:10:44.827697 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:10:44.827831 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab.
Entry for principal oryx/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab.
I20260925 05:10:44.850498 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45435
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab
--principal=oryx/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:44.933671 10813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:44.933908 10813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:44.933935 10813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:44.935092 10813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:44.935132 10813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:44.935153 10813 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:44.935173 10813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:44.936690 10813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab
--principal=oryx/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:44.936908 10813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:44.937096 10813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:44.942205 10818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:44.942176 10819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:44.942518 10813 server_base.cc:1034] running on GCE node
W20260925 05:10:44.942234 10821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:44.942791 10813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:44.943033 10813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:44.944207 10813 hybrid_clock.cc:648] HybridClock initialized: now 1790313044944183 us; error 41 us; skew 500 ppm
Sep 25 05:10:44 dist-test-slave-1lvv krb5kdc[10556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:44.946668 10813 init.cc:377] Logged in from keytab as oryx/127.3.6.130@KRBTEST.COM (short username oryx)
I20260925 05:10:44.947144 10813 webserver.cc:460] Webserver started at http://127.3.6.130:43669/ using document root <none> and password file <none>
I20260925 05:10:44.947399 10813 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:44.947470 10813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:44.947628 10813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:44.948781 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "c0e262f69c3b43fbb0a585880ab04b23"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "6b95d2653c72075b9bb12fae30740340"
server_key_iv: "39c7357859c05276b7cbf622f34cf691"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.949182 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "c0e262f69c3b43fbb0a585880ab04b23"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "6b95d2653c72075b9bb12fae30740340"
server_key_iv: "39c7357859c05276b7cbf622f34cf691"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.950662 10813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:10:44.951485 10828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:44.951699 10813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:44.951829 10813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "c0e262f69c3b43fbb0a585880ab04b23"
format_stamp: "Formatted at 2026-09-25 05:10:44 on dist-test-slave-1lvv"
server_key: "6b95d2653c72075b9bb12fae30740340"
server_key_iv: "39c7357859c05276b7cbf622f34cf691"
server_key_version: "encryptionkey@0"
I20260925 05:10:44.951927 10813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:44.961876 10813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:44.964042 10813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:44.964215 10813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:44.964865 10813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:44.964919 10813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:44.964969 10813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:44.964998 10813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:10:44 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:44.972623 10813 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:34163
I20260925 05:10:44.972759 10943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:34163 every 8 connection(s)
I20260925 05:10:44.973166 10813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:44.975621 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.965255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:45805 (local address 127.3.6.190:45435)
0925 05:10:44.965321 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:44.965323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:44.965524 (+ 201us) server_negotiation.cc:407] Connection header received
0925 05:10:44.966027 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:44.966029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:44.966044 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:44.966137 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:44.966962 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.967897 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.969055 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.969281 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.969376 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.972436 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:44.972437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:44.972438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:44.972446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:44.974272 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.974663 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.974664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.974664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.974723 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.974978 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.974979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.974980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.975130 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:44.975205 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:44.975467 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:10:44.975552 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:10:44.975720 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.965236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45435 (local address 127.3.6.130:45805)
0925 05:10:44.965381 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:44.965421 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:44.965926 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:44.966197 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:44.966201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:44.966504 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:44.966838 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.966846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.968034 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.968035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.968936 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.968940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.969390 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.969392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.969535 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.970095 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:44.970104 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:44.972318 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:44.974353 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.974358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.974360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.974587 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.974783 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.974784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.974784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.974885 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.975203 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:44.975208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:44.975332 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 05:10:44.975419 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:10:44.975551 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260925 05:10:44.976003 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 10813
I20260925 05:10:44.976136 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:44.976393 3098 external_mini_cluster.cc:1428] Setting key 41bff84f16582d71b19b05841a5e296a
I20260925 05:10:44.976414 10944 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45435
I20260925 05:10:44.976508 10944 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:44.976752 10944 heartbeater.cc:511] Master 127.3.6.190:45435 requested a full tablet report, sending...
I20260925 05:10:44.977176 10601 ts_manager.cc:194] Registered new tserver with Master: c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163)
I20260925 05:10:44.977635 10601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:45805
I20260925 05:10:44.984675 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.977626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:33197 (local address 127.3.6.190:45435)
0925 05:10:44.977675 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:44.977678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:44.978023 (+ 345us) server_negotiation.cc:407] Connection header received
0925 05:10:44.978127 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:44.978129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:44.978142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:44.978214 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:44.978604 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.979329 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.980227 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.980438 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.980541 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.981725 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:44.981726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:44.981727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:44.981735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:44.983301 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.983708 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.983710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.983711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.983830 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:44.984034 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:44.984035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:44.984035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:44.984210 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:44.984294 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:44.984472 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 05:10:44.984571 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:44.984694 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:44.977567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45435 (local address 127.3.6.130:33197)
0925 05:10:44.977948 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:44.977960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:44.978051 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:44.978238 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:44.978239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:44.978455 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:44.978514 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.978516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.979443 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.979444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.980135 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:44.980138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:44.980543 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:44.980544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:44.980615 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:44.981171 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:44.981175 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:44.981623 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:44.983396 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.983397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.983398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.983607 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.983889 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:44.983890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:44.983890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:44.983957 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:44.984297 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:44.984298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:44.984349 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 05:10:44.984440 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 05:10:44.984564 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for oryx/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab.
Entry for principal oryx/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab.
I20260925 05:10:45.008399 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45435
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab
--principal=oryx/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:45.094360 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:45.094599 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:45.094686 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:45.096021 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:45.096071 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:45.096094 10953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:10:45.096113 10953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:45.097760 10953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab
--principal=oryx/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.6.190:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:45.098048 10953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:45.098311 10953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:45.103638 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:45.103727 10953 server_base.cc:1034] running on GCE node
W20260925 05:10:45.103618 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:45.103590 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:45.104054 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:45.104396 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:45.105571 10953 hybrid_clock.cc:648] HybridClock initialized: now 1790313045105546 us; error 39 us; skew 500 ppm
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313045, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:45.108268 10953 init.cc:377] Logged in from keytab as oryx/127.3.6.131@KRBTEST.COM (short username oryx)
I20260925 05:10:45.108765 10953 webserver.cc:460] Webserver started at http://127.3.6.131:43883/ using document root <none> and password file <none>
I20260925 05:10:45.109014 10953 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:45.109084 10953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:45.109238 10953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:45.110437 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "8324c9dfbf6b44f0bad0a2db19199450"
format_stamp: "Formatted at 2026-09-25 05:10:45 on dist-test-slave-1lvv"
server_key: "1200d1a7cfcf2a8cde66563197d7966e"
server_key_iv: "fe733b48dfcbdbb1bd2902236378200b"
server_key_version: "encryptionkey@0"
I20260925 05:10:45.110831 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "8324c9dfbf6b44f0bad0a2db19199450"
format_stamp: "Formatted at 2026-09-25 05:10:45 on dist-test-slave-1lvv"
server_key: "1200d1a7cfcf2a8cde66563197d7966e"
server_key_iv: "fe733b48dfcbdbb1bd2902236378200b"
server_key_version: "encryptionkey@0"
I20260925 05:10:45.112392 10953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:10:45.113180 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:45.113353 10953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:45.113466 10953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "8324c9dfbf6b44f0bad0a2db19199450"
format_stamp: "Formatted at 2026-09-25 05:10:45 on dist-test-slave-1lvv"
server_key: "1200d1a7cfcf2a8cde66563197d7966e"
server_key_iv: "fe733b48dfcbdbb1bd2902236378200b"
server_key_version: "encryptionkey@0"
I20260925 05:10:45.113547 10953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:45.123153 10953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:45.125773 10953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:45.125958 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:45.126551 10953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:45.126614 10953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:45.126653 10953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:45.126681 10953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:45.134403 10953 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:40859
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313045, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:45.134490 11083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:40859 every 8 connection(s)
I20260925 05:10:45.134889 10953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:10:45.138149 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.126973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:48479 (local address 127.3.6.190:45435)
0925 05:10:45.127027 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.127030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.127314 (+ 284us) server_negotiation.cc:407] Connection header received
0925 05:10:45.127868 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.127870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.127885 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.127970 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:45.128619 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.129476 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.131353 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.131723 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.131838 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.135082 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:45.135085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:45.135085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:45.135095 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:45.136842 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.137280 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.137281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.137281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.137337 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.137564 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.137565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.137565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.137695 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:45.137748 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.138011 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.138080 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:45.138408 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.126974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45435 (local address 127.3.6.131:48479)
0925 05:10:45.127139 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.127207 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.127714 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.128029 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.128032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.128332 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:45.128537 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.128542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.129617 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.129618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.131214 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.131219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.131946 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.131947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.132144 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.132929 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:45.132943 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:45.134939 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:45.136934 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:45.136941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:45.136943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:45.137200 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:45.137396 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:45.137397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:45.137397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:45.137508 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:45.137752 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:45.137757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:45.137893 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.137984 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.138125 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260925 05:10:45.138989 11084 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45435
I20260925 05:10:45.139081 11084 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:45.139307 11084 heartbeater.cc:511] Master 127.3.6.190:45435 requested a full tablet report, sending...
I20260925 05:10:45.139863 10600 ts_manager.cc:194] Registered new tserver with Master: 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859)
I20260925 05:10:45.140355 10600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:48479
I20260925 05:10:45.144881 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 10953
I20260925 05:10:45.144973 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:45.145197 3098 external_mini_cluster.cc:1428] Setting key 382afb8de5e500a6f44c7c1bbdfdbc44
I20260925 05:10:45.146030 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.140314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:56291 (local address 127.3.6.190:45435)
0925 05:10:45.140368 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.140370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.140403 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:10:45.140474 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.140475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.140484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.140547 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:45.140908 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.141398 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.142151 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.142359 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.142447 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.143475 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:45.143476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:45.143477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:45.143484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:45.144805 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.145222 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.145223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.145223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.145286 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.145487 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.145488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.145488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.145608 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:45.145685 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.145864 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.145936 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:10:45.146049 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.140185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45435 (local address 127.3.6.131:56291)
0925 05:10:45.140339 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.140351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.140422 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.140555 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.140556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.140752 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:45.140816 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.140818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.141495 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.141496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.142050 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.142052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.142449 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.142450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.142523 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.143043 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:45.143055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:45.143361 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:45.144904 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:45.144905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:45.144906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:45.145142 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:45.145340 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:45.145341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:45.145341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:45.145402 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:45.145688 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:45.145689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:45.145737 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.145818 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.145936 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260925 05:10:45.146409 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313045, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:10:45.170493 10651 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:45.164454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.3.6.190:45435)
0925 05:10:45.164523 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.164525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.164534 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:45.164608 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.164609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.164617 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.164671 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:45.165275 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.165770 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.166516 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.166704 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.166759 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.170382 (+ 3623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57580: BlockingRecv error: recv got EOF from 127.0.0.1:57580 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313045, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:45.178371 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.171363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.3.6.190:45435)
0925 05:10:45.171437 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.171439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.171482 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:10:45.171525 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.171526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.171535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.171583 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:45.172128 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.172615 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.173275 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.173449 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.173526 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.175907 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:45.175909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:45.175909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:45.175916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:45.177425 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.177750 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.177751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.177751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.177805 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.177966 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.177966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.177966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.178067 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:45.178142 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.178229 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.178285 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:10:45.180325 10600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:45.182053 10600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:45.189441 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.185228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.3.6.130:34163)
0925 05:10:45.185397 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.185403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.185415 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:45.185466 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.185468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.185486 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.185597 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.185879 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.187068 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.188167 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.188931 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.189020 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.189111 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.189275 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.189326 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260925 05:10:45.189455 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.184894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:34163 (local address 127.0.0.1:58990)
0925 05:10:45.185115 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.185189 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.185262 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.185697 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.185701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.185704 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.185792 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.185794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.187215 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.187216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.188075 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.188077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.189032 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.189033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.189133 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.189153 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.189248 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.189319 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":19,"thread_start_us":74,"threads_started":1}
I20260925 05:10:45.189553 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.184951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40477 (local address 127.0.0.1:45498)
0925 05:10:45.185417 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.185430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.185486 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.185789 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.185789 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.185791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.185833 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.185835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.187120 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.187121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.188485 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.188487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.189308 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.189309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.189359 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.189362 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.189393 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.189432 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":95,"threads_started":1}
I20260925 05:10:45.189667 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.185213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.3.6.131:40859)
0925 05:10:45.185658 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.185663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.185683 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:10:45.185725 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.185727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.185744 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.185822 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.186032 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.186745 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.188285 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.188955 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.189020 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.189110 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.189504 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.189551 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":110,"threads_started":1}
I20260925 05:10:45.189463 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.184940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:40859 (local address 127.0.0.1:51040)
0925 05:10:45.185381 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.185391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.185431 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.185893 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.185893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.185895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.185947 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.185948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.186849 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.186850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.188178 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.188181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.189151 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.189153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.189253 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.189258 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.189313 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.189356 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":106,"threads_started":1}
I20260925 05:10:45.190006 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.185448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.3.6.129:40477)
0925 05:10:45.185621 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.185625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.185635 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:45.185671 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.185673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.185690 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.185907 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.185972 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.187027 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.188575 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.189215 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.189692 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.189824 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.189854 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.189901 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260925 05:10:45.190828 10738 tablet_service.cc:1467] Processing CreateTablet for tablet ee02f51dabfb490292fb5a6eadcdd7a2 (DEFAULT_TABLE table=test-table [id=5587cb0ad766494aa3afd209624805a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.190828 11018 tablet_service.cc:1467] Processing CreateTablet for tablet ee02f51dabfb490292fb5a6eadcdd7a2 (DEFAULT_TABLE table=test-table [id=5587cb0ad766494aa3afd209624805a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.190977 10878 tablet_service.cc:1467] Processing CreateTablet for tablet ee02f51dabfb490292fb5a6eadcdd7a2 (DEFAULT_TABLE table=test-table [id=5587cb0ad766494aa3afd209624805a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.191223 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee02f51dabfb490292fb5a6eadcdd7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.191223 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee02f51dabfb490292fb5a6eadcdd7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.191367 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee02f51dabfb490292fb5a6eadcdd7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.194532 11112 tablet_bootstrap.cc:492] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Bootstrap starting.
I20260925 05:10:45.195201 11113 tablet_bootstrap.cc:492] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Bootstrap starting.
I20260925 05:10:45.195266 11112 tablet_bootstrap.cc:654] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.195469 11114 tablet_bootstrap.cc:492] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Bootstrap starting.
I20260925 05:10:45.195817 11112 log.cc:824] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:45.195930 11113 tablet_bootstrap.cc:654] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.196228 11113 log.cc:824] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:45.196241 11114 tablet_bootstrap.cc:654] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.196606 11114 log.cc:824] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:45.196731 11112 tablet_bootstrap.cc:492] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: No bootstrap required, opened a new log
I20260925 05:10:45.196810 11112 ts_tablet_manager.cc:1397] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:10:45.197058 11113 tablet_bootstrap.cc:492] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: No bootstrap required, opened a new log
I20260925 05:10:45.197126 11113 ts_tablet_manager.cc:1397] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:10:45.197364 11114 tablet_bootstrap.cc:492] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: No bootstrap required, opened a new log
I20260925 05:10:45.197450 11114 ts_tablet_manager.cc:1397] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:10:45.198905 11112 raft_consensus.cc:348] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.198933 11114 raft_consensus.cc:348] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.199095 11112 raft_consensus.cc:374] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.199112 11114 raft_consensus.cc:374] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.199133 11112 raft_consensus.cc:729] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e262f69c3b43fbb0a585880ab04b23, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.199156 11114 raft_consensus.cc:729] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08251cc1c3ce4aa29a946868c131a007, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.199077 11113 raft_consensus.cc:348] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.199211 11113 raft_consensus.cc:374] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.199255 11113 raft_consensus.cc:729] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324c9dfbf6b44f0bad0a2db19199450, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.199301 11114 consensus_queue.cc:260] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.199301 11112 consensus_queue.cc:260] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.199373 11113 consensus_queue.cc:260] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.199609 11112 ts_tablet_manager.cc:1428] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:10:45.199620 11113 ts_tablet_manager.cc:1428] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:10:45.199610 11114 ts_tablet_manager.cc:1428] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:10:45.199679 11084 heartbeater.cc:503] Master 127.3.6.190:45435 was elected leader, sending a full tablet report...
I20260925 05:10:45.199784 10804 heartbeater.cc:503] Master 127.3.6.190:45435 was elected leader, sending a full tablet report...
I20260925 05:10:45.200083 10944 heartbeater.cc:503] Master 127.3.6.190:45435 was elected leader, sending a full tablet report...
I20260925 05:10:45.205627 11119 raft_consensus.cc:482] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:45.205775 11119 raft_consensus.cc:504] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.206250 11119 leader_election.cc:288] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859), 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477)
I20260925 05:10:45.209352 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.206511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:49353 (local address 127.3.6.131:40859)
0925 05:10:45.206556 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.206558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.206567 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:45.206661 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.206662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.206673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.206725 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.206877 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.207395 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.208445 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.208972 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.209056 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.209067 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.209236 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.209273 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:10:45.209352 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.206454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:40859 (local address 127.3.6.130:49353)
0925 05:10:45.206517 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.206530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.206580 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.206726 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.206727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.206728 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.206806 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.206808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.207493 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.207494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.208348 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.208351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.209053 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.209054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.209147 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.209150 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.209227 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.209275 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:10:45.209613 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.206549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:37159 (local address 127.3.6.129:40477)
0925 05:10:45.206600 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.206603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.206800 (+ 197us) server_negotiation.cc:407] Connection header received
0925 05:10:45.206889 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.206890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.206903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.206984 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.207174 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.207674 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.208786 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.209223 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.209328 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.209339 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.209527 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.209568 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:10:45.209673 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.206461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40477 (local address 127.3.6.130:37159)
0925 05:10:45.206708 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.206718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.206816 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.206988 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.206989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.206990 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.207093 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.207095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.207807 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.207808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.208693 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.208696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.209332 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.209333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.209431 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.209437 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.209519 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.209576 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":64,"thread_start_us":130,"threads_started":1}
I20260925 05:10:45.210227 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02f51dabfb490292fb5a6eadcdd7a2" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324c9dfbf6b44f0bad0a2db19199450" is_pre_election: true
I20260925 05:10:45.210199 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02f51dabfb490292fb5a6eadcdd7a2" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007" is_pre_election: true
I20260925 05:10:45.210376 10758 raft_consensus.cc:2454] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 0.
I20260925 05:10:45.210376 11038 raft_consensus.cc:2454] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 0.
I20260925 05:10:45.210637 10830 leader_election.cc:302] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08251cc1c3ce4aa29a946868c131a007, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.210765 11119 raft_consensus.cc:2781] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:45.210825 11119 raft_consensus.cc:482] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:45.210868 11119 raft_consensus.cc:3037] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.211918 11119 raft_consensus.cc:504] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.212072 11119 leader_election.cc:288] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 election: Requested vote from peers 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859), 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477)
I20260925 05:10:45.212206 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02f51dabfb490292fb5a6eadcdd7a2" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007"
I20260925 05:10:45.212293 10758 raft_consensus.cc:3037] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.212282 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee02f51dabfb490292fb5a6eadcdd7a2" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324c9dfbf6b44f0bad0a2db19199450"
I20260925 05:10:45.212409 11038 raft_consensus.cc:3037] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.213253 11038 raft_consensus.cc:2454] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 1.
I20260925 05:10:45.213328 10758 raft_consensus.cc:2454] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 1.
I20260925 05:10:45.213496 10831 leader_election.cc:302] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8324c9dfbf6b44f0bad0a2db19199450, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.213621 11119 raft_consensus.cc:2781] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:45.213809 11119 raft_consensus.cc:686] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 LEADER]: Becoming Leader. State: Replica: c0e262f69c3b43fbb0a585880ab04b23, State: Running, Role: LEADER
I20260925 05:10:45.213922 11119 consensus_queue.cc:237] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } }
I20260925 05:10:45.214824 10599 catalog_manager.cc:5495] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130). New cstate: current_term: 1 leader_uuid: "c0e262f69c3b43fbb0a585880ab04b23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } health_report { overall_health: UNKNOWN } } }
W20260925 05:10:45.224128 10945 tablet.cc:2369] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:45.228547 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.226098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.3.6.130:34163)
0925 05:10:45.226164 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.226166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.226176 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:45.226214 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.226215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.226234 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.226301 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.226438 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.226898 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.227704 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.227970 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.228036 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.228240 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.228342 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.228404 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.228417 (+ 13us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.228436 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:10:45.230485 10758 raft_consensus.cc:1261] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Refusing update from remote peer c0e262f69c3b43fbb0a585880ab04b23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.230489 11038 raft_consensus.cc:1261] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Refusing update from remote peer c0e262f69c3b43fbb0a585880ab04b23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.230830 11119 consensus_queue.cc:1035] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.230978 11122 consensus_queue.cc:1035] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.233597 11125 mvcc.cc:205] Tried to move back new op lower bound from 7333122233261199360 to 7333122233197043712. Current Snapshot: MvccSnapshot[applied={T|T < 7333122233261199360}]
I20260925 05:10:45.233762 11126 mvcc.cc:205] Tried to move back new op lower bound from 7333122233261199360 to 7333122233197043712. Current Snapshot: MvccSnapshot[applied={T|T < 7333122233261199360}]
I20260925 05:10:45.234166 11127 mvcc.cc:205] Tried to move back new op lower bound from 7333122233261199360 to 7333122233197043712. Current Snapshot: MvccSnapshot[applied={T|T < 7333122233261199360}]
I20260925 05:10:45.236629 10599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.236743 10599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.237571 10599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Sending DeleteTablet for 3 replicas of tablet ee02f51dabfb490292fb5a6eadcdd7a2
I20260925 05:10:45.237903 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet ee02f51dabfb490292fb5a6eadcdd7a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58990
I20260925 05:10:45.237965 11018 tablet_service.cc:1514] Processing DeleteTablet for tablet ee02f51dabfb490292fb5a6eadcdd7a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:51040
I20260925 05:10:45.238118 10738 tablet_service.cc:1514] Processing DeleteTablet for tablet ee02f51dabfb490292fb5a6eadcdd7a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45498
I20260925 05:10:45.238368 11136 tablet_replica.cc:330] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: stopping tablet replica
I20260925 05:10:45.238379 11135 tablet_replica.cc:330] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: stopping tablet replica
I20260925 05:10:45.238416 10599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:10:45.238521 11135 raft_consensus.cc:2229] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 LEADER]: Raft consensus shutting down.
W20260925 05:10:45.238580 10599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:45.238670 11135 raft_consensus.cc:2258] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.238826 11136 raft_consensus.cc:2229] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.238989 11136 raft_consensus.cc:2258] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.239482 11135 ts_tablet_manager.cc:1905] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.240067 11137 tablet_replica.cc:330] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: stopping tablet replica
I20260925 05:10:45.240212 11137 raft_consensus.cc:2229] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.240284 10878 tablet_service.cc:1467] Processing CreateTablet for tablet b2ab011c56114df29f518bbbe9782a2f (DEFAULT_TABLE table=test-table [id=9f0a411482e44124b8e3403908b85b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.240351 11137 raft_consensus.cc:2258] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.240795 11018 tablet_service.cc:1467] Processing CreateTablet for tablet b2ab011c56114df29f518bbbe9782a2f (DEFAULT_TABLE table=test-table [id=9f0a411482e44124b8e3403908b85b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.240875 11137 ts_tablet_manager.cc:1905] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.240928 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ab011c56114df29f518bbbe9782a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.242547 11113 tablet_bootstrap.cc:492] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Bootstrap starting.
I20260925 05:10:45.242699 11137 ts_tablet_manager.cc:1918] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:45.242775 11137 log.cc:1196] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/ee02f51dabfb490292fb5a6eadcdd7a2
I20260925 05:10:45.242900 10738 tablet_service.cc:1467] Processing CreateTablet for tablet b2ab011c56114df29f518bbbe9782a2f (DEFAULT_TABLE table=test-table [id=9f0a411482e44124b8e3403908b85b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.242993 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ab011c56114df29f518bbbe9782a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.243062 11137 ts_tablet_manager.cc:1939] T ee02f51dabfb490292fb5a6eadcdd7a2 P 08251cc1c3ce4aa29a946868c131a007: Deleting consensus metadata
I20260925 05:10:45.243055 11113 tablet_bootstrap.cc:654] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.243459 10586 catalog_manager.cc:4841] TS 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477): tablet ee02f51dabfb490292fb5a6eadcdd7a2 (table test-table [id=5587cb0ad766494aa3afd209624805a4]) successfully deleted
I20260925 05:10:45.244213 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ab011c56114df29f518bbbe9782a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.244321 11114 tablet_bootstrap.cc:492] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Bootstrap starting.
I20260925 05:10:45.244647 11114 tablet_bootstrap.cc:654] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.245219 11114 tablet_bootstrap.cc:492] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: No bootstrap required, opened a new log
I20260925 05:10:45.245261 11114 ts_tablet_manager.cc:1397] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:45.245424 11114 raft_consensus.cc:348] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.245491 11114 raft_consensus.cc:374] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.245509 11114 raft_consensus.cc:729] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08251cc1c3ce4aa29a946868c131a007, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.245555 11114 consensus_queue.cc:260] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.245664 11114 ts_tablet_manager.cc:1428] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:45.245718 11112 tablet_bootstrap.cc:492] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: Bootstrap starting.
I20260925 05:10:45.246163 11112 tablet_bootstrap.cc:654] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.246415 11135 ts_tablet_manager.cc:1918] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:45.246505 11135 log.cc:1196] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/ee02f51dabfb490292fb5a6eadcdd7a2
I20260925 05:10:45.246809 11135 ts_tablet_manager.cc:1939] T ee02f51dabfb490292fb5a6eadcdd7a2 P c0e262f69c3b43fbb0a585880ab04b23: Deleting consensus metadata
I20260925 05:10:45.247160 11112 tablet_bootstrap.cc:492] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: No bootstrap required, opened a new log
I20260925 05:10:45.247224 11112 ts_tablet_manager.cc:1397] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:10:45.247287 10588 catalog_manager.cc:4841] TS c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163): tablet ee02f51dabfb490292fb5a6eadcdd7a2 (table test-table [id=5587cb0ad766494aa3afd209624805a4]) successfully deleted
I20260925 05:10:45.247370 11112 raft_consensus.cc:348] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.247433 11112 raft_consensus.cc:374] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.247452 11112 raft_consensus.cc:729] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e262f69c3b43fbb0a585880ab04b23, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.247501 11112 consensus_queue.cc:260] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.247591 11112 ts_tablet_manager.cc:1428] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:45.248009 11136 ts_tablet_manager.cc:1905] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.249697 11136 ts_tablet_manager.cc:1918] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:45.249787 11136 log.cc:1196] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/ee02f51dabfb490292fb5a6eadcdd7a2
I20260925 05:10:45.250046 11136 ts_tablet_manager.cc:1939] T ee02f51dabfb490292fb5a6eadcdd7a2 P 8324c9dfbf6b44f0bad0a2db19199450: Deleting consensus metadata
I20260925 05:10:45.250526 10587 catalog_manager.cc:4841] TS 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859): tablet ee02f51dabfb490292fb5a6eadcdd7a2 (table test-table [id=5587cb0ad766494aa3afd209624805a4]) successfully deleted
I20260925 05:10:45.252175 11113 tablet_bootstrap.cc:492] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: No bootstrap required, opened a new log
I20260925 05:10:45.252239 11113 ts_tablet_manager.cc:1397] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 05:10:45.252403 11113 raft_consensus.cc:348] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.252485 11113 raft_consensus.cc:374] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.252504 11113 raft_consensus.cc:729] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324c9dfbf6b44f0bad0a2db19199450, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.252550 11113 consensus_queue.cc:260] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.252646 11113 ts_tablet_manager.cc:1428] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:45.253933 11120 raft_consensus.cc:482] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:45.254025 11120 raft_consensus.cc:504] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.254376 11120 leader_election.cc:288] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477), c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163)
I20260925 05:10:45.258246 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.254553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:34163 (local address 127.3.6.131:55777)
0925 05:10:45.254610 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.254624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.254668 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.254891 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.254893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.254895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.254968 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.254970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.255894 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.255896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.257155 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.257158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.257976 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.257977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.258039 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.258043 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.258112 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.258163 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:10:45.258260 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.254657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:55777 (local address 127.3.6.130:34163)
0925 05:10:45.254705 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.254707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.254718 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:45.254747 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.254748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.254773 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.254835 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.255046 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.255813 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.257281 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.257882 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.257972 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.257984 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.258138 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.258201 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:10:45.258821 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ab011c56114df29f518bbbe9782a2f" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e262f69c3b43fbb0a585880ab04b23" is_pre_election: true
I20260925 05:10:45.258996 10898 raft_consensus.cc:2454] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 0.
I20260925 05:10:45.259027 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.254616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40477 (local address 127.3.6.131:50833)
0925 05:10:45.254883 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.254894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.254952 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.255074 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.255075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.255076 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.255153 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.255154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.256095 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.256096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.257182 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.257183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.258749 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.258750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.258836 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.258841 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.258896 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.258939 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":96,"threads_started":1}
I20260925 05:10:45.259048 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.254833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:50833 (local address 127.3.6.129:40477)
0925 05:10:45.254879 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.254881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.254940 (+ 59us) server_negotiation.cc:407] Connection header received
0925 05:10:45.255006 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.255007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.255016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.255073 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.255207 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.256000 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.258034 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.258613 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.258697 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.258708 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.258922 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.258976 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:10:45.259436 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ab011c56114df29f518bbbe9782a2f" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007" is_pre_election: true
I20260925 05:10:45.259480 10972 leader_election.cc:302] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8324c9dfbf6b44f0bad0a2db19199450, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.259538 10758 raft_consensus.cc:2454] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 0.
I20260925 05:10:45.259642 11120 raft_consensus.cc:2781] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:45.259692 11120 raft_consensus.cc:482] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:45.259714 11120 raft_consensus.cc:3037] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.260655 11120 raft_consensus.cc:504] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.260823 11120 leader_election.cc:288] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 election: Requested vote from peers 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477), c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163)
I20260925 05:10:45.261014 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ab011c56114df29f518bbbe9782a2f" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e262f69c3b43fbb0a585880ab04b23"
I20260925 05:10:45.261037 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ab011c56114df29f518bbbe9782a2f" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007"
I20260925 05:10:45.261102 10758 raft_consensus.cc:3037] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.261102 10898 raft_consensus.cc:3037] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.261932 10898 raft_consensus.cc:2454] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 1.
I20260925 05:10:45.261932 10758 raft_consensus.cc:2454] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 1.
I20260925 05:10:45.262131 10970 leader_election.cc:302] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08251cc1c3ce4aa29a946868c131a007, 8324c9dfbf6b44f0bad0a2db19199450; no voters:
I20260925 05:10:45.262255 11120 raft_consensus.cc:2781] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:45.262446 11120 raft_consensus.cc:686] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 LEADER]: Becoming Leader. State: Replica: 8324c9dfbf6b44f0bad0a2db19199450, State: Running, Role: LEADER
I20260925 05:10:45.262526 11120 consensus_queue.cc:237] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.263381 10599 catalog_manager.cc:5495] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } health_report { overall_health: HEALTHY } } }
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313044, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.190@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:45.288597 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.280696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.3.6.190:45435)
0925 05:10:45.280772 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.280774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.280924 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:10:45.280963 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.280964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.280976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.281050 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:45.281507 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.282296 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.283023 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.283345 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.283423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.285840 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:45.285842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:45.285842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:45.285848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:45.287319 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.287834 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.287835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.287835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.287888 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.288091 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.288091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.288092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.288215 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:45.288266 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.288451 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.288526 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:10:45.288743 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.280631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45435 (local address 127.0.0.1:57596)
0925 05:10:45.280842 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.280852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.280917 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.281072 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.281072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.281380 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:45.281428 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.281430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.282385 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.282386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.282926 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.282928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.283431 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.283432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.283474 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.284128 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:45.284133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:45.285713 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:45.287409 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:45.287419 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:45.287422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:45.287721 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:45.287950 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:45.287951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:45.287952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:45.288027 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:45.288279 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:45.288288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:45.288349 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.288426 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.288626 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":67,"thread_start_us":103,"threads_started":1}
I20260925 05:10:45.291126 10599 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:57596:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:10:45.291342 10599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:45.292912 11018 tablet_service.cc:1467] Processing CreateTablet for tablet d85e2ece7dce422dbf0a4b7abbe81263 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=085e8c30fb494b3dbc7158bce2cb9f85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:45.292889 10738 tablet_service.cc:1467] Processing CreateTablet for tablet d85e2ece7dce422dbf0a4b7abbe81263 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=085e8c30fb494b3dbc7158bce2cb9f85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:45.292917 10878 tablet_service.cc:1467] Processing CreateTablet for tablet d85e2ece7dce422dbf0a4b7abbe81263 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=085e8c30fb494b3dbc7158bce2cb9f85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:10:45.293071 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85e2ece7dce422dbf0a4b7abbe81263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.293071 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85e2ece7dce422dbf0a4b7abbe81263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.293071 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85e2ece7dce422dbf0a4b7abbe81263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.294616 11112 tablet_bootstrap.cc:492] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: Bootstrap starting.
I20260925 05:10:45.294627 11113 tablet_bootstrap.cc:492] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: Bootstrap starting.
I20260925 05:10:45.294627 11114 tablet_bootstrap.cc:492] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: Bootstrap starting.
I20260925 05:10:45.295055 11112 tablet_bootstrap.cc:654] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.295060 11113 tablet_bootstrap.cc:654] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.295060 11114 tablet_bootstrap.cc:654] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.295970 11113 tablet_bootstrap.cc:492] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: No bootstrap required, opened a new log
I20260925 05:10:45.296026 11113 ts_tablet_manager.cc:1397] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:45.296087 11112 tablet_bootstrap.cc:492] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: No bootstrap required, opened a new log
I20260925 05:10:45.296131 11112 ts_tablet_manager.cc:1397] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:10:45.296135 11114 tablet_bootstrap.cc:492] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: No bootstrap required, opened a new log
I20260925 05:10:45.296176 11114 ts_tablet_manager.cc:1397] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:10:45.296272 11112 raft_consensus.cc:348] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.296341 11112 raft_consensus.cc:374] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.296362 11112 raft_consensus.cc:729] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e262f69c3b43fbb0a585880ab04b23, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.296326 11114 raft_consensus.cc:348] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.296408 11114 raft_consensus.cc:374] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.296404 11112 consensus_queue.cc:260] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.296430 11114 raft_consensus.cc:729] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08251cc1c3ce4aa29a946868c131a007, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.296494 11112 ts_tablet_manager.cc:1428] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:45.296489 11114 consensus_queue.cc:260] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.296485 11113 raft_consensus.cc:348] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.296552 11113 raft_consensus.cc:374] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.296574 11113 raft_consensus.cc:729] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324c9dfbf6b44f0bad0a2db19199450, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.296576 11114 ts_tablet_manager.cc:1428] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:45.296581 11119 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.296617 11113 consensus_queue.cc:260] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.296641 11119 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:45.296710 11113 ts_tablet_manager.cc:1428] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:10:45.296749 11118 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.296826 11118 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:45.296842 11120 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.296921 11120 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 05:10:45.316663 10805 tablet.cc:2369] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:45.376964 11120 raft_consensus.cc:482] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:45.377060 11120 raft_consensus.cc:504] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.377247 11120 leader_election.cc:288] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477), c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163)
I20260925 05:10:45.377478 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85e2ece7dce422dbf0a4b7abbe81263" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007" is_pre_election: true
I20260925 05:10:45.377511 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85e2ece7dce422dbf0a4b7abbe81263" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e262f69c3b43fbb0a585880ab04b23" is_pre_election: true
I20260925 05:10:45.377610 10758 raft_consensus.cc:2454] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 0.
I20260925 05:10:45.377610 10898 raft_consensus.cc:2454] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 0.
I20260925 05:10:45.377758 10972 leader_election.cc:302] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8324c9dfbf6b44f0bad0a2db19199450, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.377902 11120 raft_consensus.cc:2781] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:45.377954 11120 raft_consensus.cc:482] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:45.377969 11120 raft_consensus.cc:3037] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.378664 11120 raft_consensus.cc:504] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.378840 11120 leader_election.cc:288] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 election: Requested vote from peers 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477), c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163)
I20260925 05:10:45.378995 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85e2ece7dce422dbf0a4b7abbe81263" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007"
I20260925 05:10:45.379017 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85e2ece7dce422dbf0a4b7abbe81263" candidate_uuid: "8324c9dfbf6b44f0bad0a2db19199450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e262f69c3b43fbb0a585880ab04b23"
I20260925 05:10:45.379076 10758 raft_consensus.cc:3037] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.379093 10898 raft_consensus.cc:3037] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.379912 10898 raft_consensus.cc:2454] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 1.
I20260925 05:10:45.379915 10758 raft_consensus.cc:2454] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8324c9dfbf6b44f0bad0a2db19199450 in term 1.
I20260925 05:10:45.380100 10972 leader_election.cc:302] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8324c9dfbf6b44f0bad0a2db19199450, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.380204 11120 raft_consensus.cc:2781] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:45.380262 11120 raft_consensus.cc:686] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 LEADER]: Becoming Leader. State: Replica: 8324c9dfbf6b44f0bad0a2db19199450, State: Running, Role: LEADER
I20260925 05:10:45.380312 11120 consensus_queue.cc:237] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.380690 11142 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: TxnStatusTablet state changed. Reason: New leader 8324c9dfbf6b44f0bad0a2db19199450. Latest consensus state: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.380798 11142 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:45.381022 11154 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:10:45.381158 11154 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:10:45.381330 10599 catalog_manager.cc:5495] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } health_report { overall_health: HEALTHY } } }
W20260925 05:10:45.385601 11085 tablet.cc:2369] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:10:45.394740 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.391938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.3.6.131:40859)
0925 05:10:45.391981 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.391984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.392016 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:10:45.392115 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.392116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.392133 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.392206 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.392343 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.392828 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.393961 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.394149 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.394235 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.394429 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.394528 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.394584 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.394658 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.394677 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:10:45.394794 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.391878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:40859 (local address 127.0.0.1:51052)
0925 05:10:45.391970 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.391983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.392036 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.392214 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.392216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.392217 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:45.392276 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.392279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.392911 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.392912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.393864 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.393867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.394243 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.394245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.394316 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.394335 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.394593 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.394601 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.394656 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.394708 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 05:10:45.395995 10758 raft_consensus.cc:1261] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Refusing update from remote peer 8324c9dfbf6b44f0bad0a2db19199450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.396123 10898 raft_consensus.cc:1261] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Refusing update from remote peer 8324c9dfbf6b44f0bad0a2db19199450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.396317 11142 consensus_queue.cc:1035] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.396431 11120 consensus_queue.cc:1035] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.397022 11118 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: TxnStatusTablet state changed. Reason: New leader 8324c9dfbf6b44f0bad0a2db19199450. Latest consensus state: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.397125 11118 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:45.397475 11119 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: TxnStatusTablet state changed. Reason: New leader 8324c9dfbf6b44f0bad0a2db19199450. Latest consensus state: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.397585 11119 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:45.398080 11120 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.398164 11120 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:45.398967 11118 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.399060 11118 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P 08251cc1c3ce4aa29a946868c131a007: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:45.399302 11142 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.399397 11142 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P 8324c9dfbf6b44f0bad0a2db19199450: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:10:45.400125 11119 tablet_replica.cc:439] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8324c9dfbf6b44f0bad0a2db19199450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } } }
I20260925 05:10:45.400234 11119 tablet_replica.cc:442] T d85e2ece7dce422dbf0a4b7abbe81263 P c0e262f69c3b43fbb0a585880ab04b23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:10:45.402240 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.399582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.3.6.131:40859)
0925 05:10:45.399627 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.399629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.399637 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:10:45.399723 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.399725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.399735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.399853 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.400017 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.400756 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.401525 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.401737 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.401818 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.401972 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.402006 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.402061 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.402127 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.402146 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:10:45.407662 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.405304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:56565 (local address 127.3.6.131:40859)
0925 05:10:45.405342 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.405344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.405373 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:10:45.405462 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.405463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.405472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.405527 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.405674 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.406138 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.406935 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.407120 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.407219 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.407393 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.407429 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.407490 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.407548 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.407572 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":19}
I20260925 05:10:45.407662 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.405240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:40859 (local address 127.3.6.131:56565)
0925 05:10:45.405322 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.405336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.405389 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.405530 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.405530 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.405531 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:45.405608 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.405609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.406244 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.406245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.406835 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.406837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.407217 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.407218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.407285 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.407299 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.407489 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.407491 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.407546 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.407592 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
I20260925 05:10:45.410156 10898 raft_consensus.cc:1261] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Refusing update from remote peer 8324c9dfbf6b44f0bad0a2db19199450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.410198 10758 raft_consensus.cc:1261] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Refusing update from remote peer 8324c9dfbf6b44f0bad0a2db19199450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.410429 11120 consensus_queue.cc:1035] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.410555 11142 consensus_queue.cc:1035] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.432211 10600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.432331 10600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.433018 10600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Sending DeleteTablet for 3 replicas of tablet b2ab011c56114df29f518bbbe9782a2f
I20260925 05:10:45.433315 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ab011c56114df29f518bbbe9782a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58990
I20260925 05:10:45.433329 10738 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ab011c56114df29f518bbbe9782a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45498
I20260925 05:10:45.433339 11018 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ab011c56114df29f518bbbe9782a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:51040
I20260925 05:10:45.433422 11135 tablet_replica.cc:330] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: stopping tablet replica
I20260925 05:10:45.433459 11136 tablet_replica.cc:330] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: stopping tablet replica
I20260925 05:10:45.433457 11137 tablet_replica.cc:330] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: stopping tablet replica
I20260925 05:10:45.433494 11135 raft_consensus.cc:2229] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.433509 11136 raft_consensus.cc:2229] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:45.433517 11137 raft_consensus.cc:2229] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.433533 11135 raft_consensus.cc:2258] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.433557 11137 raft_consensus.cc:2258] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.433614 11136 raft_consensus.cc:2258] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.433949 11135 ts_tablet_manager.cc:1905] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.434000 11136 ts_tablet_manager.cc:1905] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.433996 11137 ts_tablet_manager.cc:1905] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.435997 11137 ts_tablet_manager.cc:1918] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:45.436005 11136 ts_tablet_manager.cc:1918] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:45.436007 11135 ts_tablet_manager.cc:1918] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:45.436069 11135 log.cc:1196] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/b2ab011c56114df29f518bbbe9782a2f
I20260925 05:10:45.436069 11136 log.cc:1196] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/b2ab011c56114df29f518bbbe9782a2f
I20260925 05:10:45.436070 11137 log.cc:1196] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/b2ab011c56114df29f518bbbe9782a2f
I20260925 05:10:45.436317 11136 ts_tablet_manager.cc:1939] T b2ab011c56114df29f518bbbe9782a2f P 8324c9dfbf6b44f0bad0a2db19199450: Deleting consensus metadata
I20260925 05:10:45.436318 11135 ts_tablet_manager.cc:1939] T b2ab011c56114df29f518bbbe9782a2f P c0e262f69c3b43fbb0a585880ab04b23: Deleting consensus metadata
I20260925 05:10:45.436347 11137 ts_tablet_manager.cc:1939] T b2ab011c56114df29f518bbbe9782a2f P 08251cc1c3ce4aa29a946868c131a007: Deleting consensus metadata
I20260925 05:10:45.436823 10586 catalog_manager.cc:4841] TS 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477): tablet b2ab011c56114df29f518bbbe9782a2f (table test-table [id=9f0a411482e44124b8e3403908b85b07]) successfully deleted
I20260925 05:10:45.439193 10587 catalog_manager.cc:4841] TS 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859): tablet b2ab011c56114df29f518bbbe9782a2f (table test-table [id=9f0a411482e44124b8e3403908b85b07]) successfully deleted
I20260925 05:10:45.439301 10588 catalog_manager.cc:4841] TS c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163): tablet b2ab011c56114df29f518bbbe9782a2f (table test-table [id=9f0a411482e44124b8e3403908b85b07]) successfully deleted
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:10:45.463238 10651 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:45.456899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.3.6.190:45435)
0925 05:10:45.456970 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.456972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.457088 (+ 116us) server_negotiation.cc:407] Connection header received
0925 05:10:45.457157 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.457158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.457167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.457244 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:45.458028 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.458530 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.459642 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.459888 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.459944 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.463146 (+ 3202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57612: BlockingRecv error: recv got EOF from 127.0.0.1:57612 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[10556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM
I20260925 05:10:45.472474 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.464243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.3.6.190:45435)
0925 05:10:45.464293 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.464296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.464475 (+ 179us) server_negotiation.cc:407] Connection header received
0925 05:10:45.464535 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.464536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.464546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.464593 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:45.465192 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.465662 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.466732 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.466987 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.467055 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.469719 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:45.469721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:45.469721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:45.469729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:45.471162 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.471531 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.471533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.471533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.471599 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:45.471869 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:45.471870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:45.471870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:45.472001 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:45.472117 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.472265 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.472359 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:10:45.474092 10600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:45.474289 10600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:45.475921 10878 tablet_service.cc:1467] Processing CreateTablet for tablet 7e704bfd32e3406da8a1a402f3cfb898 (DEFAULT_TABLE table=test-table [id=ece5e04131a54de98239a0c237605b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.475971 10738 tablet_service.cc:1467] Processing CreateTablet for tablet 7e704bfd32e3406da8a1a402f3cfb898 (DEFAULT_TABLE table=test-table [id=ece5e04131a54de98239a0c237605b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.476100 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e704bfd32e3406da8a1a402f3cfb898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.476114 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e704bfd32e3406da8a1a402f3cfb898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.477198 11018 tablet_service.cc:1467] Processing CreateTablet for tablet 7e704bfd32e3406da8a1a402f3cfb898 (DEFAULT_TABLE table=test-table [id=ece5e04131a54de98239a0c237605b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.477341 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e704bfd32e3406da8a1a402f3cfb898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.477612 11114 tablet_bootstrap.cc:492] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: Bootstrap starting.
I20260925 05:10:45.478087 11114 tablet_bootstrap.cc:654] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.478467 11112 tablet_bootstrap.cc:492] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: Bootstrap starting.
I20260925 05:10:45.478564 11113 tablet_bootstrap.cc:492] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: Bootstrap starting.
I20260925 05:10:45.478835 11112 tablet_bootstrap.cc:654] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.478991 11114 tablet_bootstrap.cc:492] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: No bootstrap required, opened a new log
I20260925 05:10:45.479002 11113 tablet_bootstrap.cc:654] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.479058 11114 ts_tablet_manager.cc:1397] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:45.479231 11114 raft_consensus.cc:348] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.479321 11114 raft_consensus.cc:374] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.479343 11114 raft_consensus.cc:729] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08251cc1c3ce4aa29a946868c131a007, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.479384 11114 consensus_queue.cc:260] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.479493 11114 ts_tablet_manager.cc:1428] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:45.479631 11112 tablet_bootstrap.cc:492] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: No bootstrap required, opened a new log
I20260925 05:10:45.479676 11112 ts_tablet_manager.cc:1397] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:45.479830 11112 raft_consensus.cc:348] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.479914 11112 raft_consensus.cc:374] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.479936 11112 raft_consensus.cc:729] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e262f69c3b43fbb0a585880ab04b23, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.479977 11112 consensus_queue.cc:260] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.480074 11112 ts_tablet_manager.cc:1428] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:45.484254 11113 tablet_bootstrap.cc:492] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: No bootstrap required, opened a new log
I20260925 05:10:45.484323 11113 ts_tablet_manager.cc:1397] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 05:10:45.484493 11113 raft_consensus.cc:348] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.484580 11113 raft_consensus.cc:374] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.484606 11113 raft_consensus.cc:729] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324c9dfbf6b44f0bad0a2db19199450, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.484655 11113 consensus_queue.cc:260] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.484760 11113 ts_tablet_manager.cc:1428] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:45.555927 11118 raft_consensus.cc:482] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:45.556052 11118 raft_consensus.cc:504] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.556463 11118 leader_election.cc:288] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163), 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859)
I20260925 05:10:45.560457 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.556791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:38403 (local address 127.3.6.131:40859)
0925 05:10:45.556836 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.556838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.556890 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:10:45.557006 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.557007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.557018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.557076 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.557246 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.558018 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.559424 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.560026 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.560105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.560118 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.560332 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.560380 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:10:45.560459 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.556655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:40859 (local address 127.3.6.129:38403)
0925 05:10:45.556820 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.556831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.556923 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.557080 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.557081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.557083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.557167 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.557169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.558141 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.558142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.559309 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.559312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.560118 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.560119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.560209 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.560215 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.560311 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.560358 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260925 05:10:45.560554 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.556777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:58109 (local address 127.3.6.130:34163)
0925 05:10:45.556829 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.556832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.556960 (+ 128us) server_negotiation.cc:407] Connection header received
0925 05:10:45.557044 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.557045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.557055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.557105 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:10:45.557314 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.558001 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.559559 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.560150 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.560241 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.560251 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.560425 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.560471 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:10:45.560990 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.556664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:34163 (local address 127.3.6.129:58109)
0925 05:10:45.556892 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.556903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.556974 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.557153 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.557154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.557156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:10:45.557249 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.557251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.558117 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.558118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.559456 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.559460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.560245 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.560246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.560332 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.560338 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.560872 (+ 534us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.560920 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":68,"threads_started":1}
I20260925 05:10:45.560958 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e704bfd32e3406da8a1a402f3cfb898" candidate_uuid: "08251cc1c3ce4aa29a946868c131a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324c9dfbf6b44f0bad0a2db19199450" is_pre_election: true
I20260925 05:10:45.561055 11038 raft_consensus.cc:2454] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08251cc1c3ce4aa29a946868c131a007 in term 0.
I20260925 05:10:45.561277 10691 leader_election.cc:302] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08251cc1c3ce4aa29a946868c131a007, 8324c9dfbf6b44f0bad0a2db19199450; no voters:
I20260925 05:10:45.561429 11118 raft_consensus.cc:2781] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:45.561484 11118 raft_consensus.cc:482] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:45.561506 11118 raft_consensus.cc:3037] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.561460 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e704bfd32e3406da8a1a402f3cfb898" candidate_uuid: "08251cc1c3ce4aa29a946868c131a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e262f69c3b43fbb0a585880ab04b23" is_pre_election: true
I20260925 05:10:45.561539 10898 raft_consensus.cc:2454] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08251cc1c3ce4aa29a946868c131a007 in term 0.
I20260925 05:10:45.562282 11118 raft_consensus.cc:504] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.562434 11118 leader_election.cc:288] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [CANDIDATE]: Term 1 election: Requested vote from peers c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163), 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859)
I20260925 05:10:45.562574 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e704bfd32e3406da8a1a402f3cfb898" candidate_uuid: "08251cc1c3ce4aa29a946868c131a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e262f69c3b43fbb0a585880ab04b23"
I20260925 05:10:45.562597 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e704bfd32e3406da8a1a402f3cfb898" candidate_uuid: "08251cc1c3ce4aa29a946868c131a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324c9dfbf6b44f0bad0a2db19199450"
I20260925 05:10:45.562642 10898 raft_consensus.cc:3037] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.562671 11038 raft_consensus.cc:3037] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.563510 10898 raft_consensus.cc:2454] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08251cc1c3ce4aa29a946868c131a007 in term 1.
I20260925 05:10:45.563634 11038 raft_consensus.cc:2454] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08251cc1c3ce4aa29a946868c131a007 in term 1.
I20260925 05:10:45.563688 10692 leader_election.cc:302] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08251cc1c3ce4aa29a946868c131a007, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.563793 11118 raft_consensus.cc:2781] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:45.563899 11118 raft_consensus.cc:686] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 1 LEADER]: Becoming Leader. State: Replica: 08251cc1c3ce4aa29a946868c131a007, State: Running, Role: LEADER
I20260925 05:10:45.563975 11118 consensus_queue.cc:237] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.564831 10600 catalog_manager.cc:5495] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "08251cc1c3ce4aa29a946868c131a007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:45.576675 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.573786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.3.6.129:40477)
0925 05:10:45.573833 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.573840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.573852 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:45.573915 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.573916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.573935 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.573995 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.574145 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.574680 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.575843 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.576029 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.576084 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.576291 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.576402 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.576462 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.576528 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.576545 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:45.577749 11038 raft_consensus.cc:1261] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Refusing update from remote peer 08251cc1c3ce4aa29a946868c131a007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.577749 10898 raft_consensus.cc:1261] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Refusing update from remote peer 08251cc1c3ce4aa29a946868c131a007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.578050 11183 consensus_queue.cc:1035] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.578228 11118 consensus_queue.cc:1035] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.582228 10600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.582333 10600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.583076 10600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Sending DeleteTablet for 3 replicas of tablet 7e704bfd32e3406da8a1a402f3cfb898
I20260925 05:10:45.583374 11018 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e704bfd32e3406da8a1a402f3cfb898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:51040
I20260925 05:10:45.583480 11136 tablet_replica.cc:330] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: stopping tablet replica
I20260925 05:10:45.583539 11136 raft_consensus.cc:2229] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.583606 11136 raft_consensus.cc:2258] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.583870 10738 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e704bfd32e3406da8a1a402f3cfb898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45498
I20260925 05:10:45.583889 10737 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e704bfd32e3406da8a1a402f3cfb898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45498
I20260925 05:10:45.583974 11137 tablet_replica.cc:330] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: stopping tablet replica
I20260925 05:10:45.584018 11137 raft_consensus.cc:2229] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:45.584110 11137 raft_consensus.cc:2258] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.584369 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e704bfd32e3406da8a1a402f3cfb898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58990
I20260925 05:10:45.584468 11135 tablet_replica.cc:330] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: stopping tablet replica
I20260925 05:10:45.584520 11135 raft_consensus.cc:2229] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.584563 11135 raft_consensus.cc:2258] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.584767 11137 ts_tablet_manager.cc:1905] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.584829 10600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:10:45.584980 10600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:10:45.586584 11018 tablet_service.cc:1467] Processing CreateTablet for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a (DEFAULT_TABLE table=test-table [id=23b54b6c8ffb404b9f5eb6a48d4fc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.586620 11136 ts_tablet_manager.cc:1905] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.586717 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.588230 11136 ts_tablet_manager.cc:1918] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:45.588263 11113 tablet_bootstrap.cc:492] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: Bootstrap starting.
I20260925 05:10:45.588289 11136 log.cc:1196] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/7e704bfd32e3406da8a1a402f3cfb898
I20260925 05:10:45.588531 11136 ts_tablet_manager.cc:1939] T 7e704bfd32e3406da8a1a402f3cfb898 P 8324c9dfbf6b44f0bad0a2db19199450: Deleting consensus metadata
I20260925 05:10:45.588605 11113 tablet_bootstrap.cc:654] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.588783 11137 ts_tablet_manager.cc:1918] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:45.588840 11137 log.cc:1196] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/7e704bfd32e3406da8a1a402f3cfb898
I20260925 05:10:45.589053 11137 ts_tablet_manager.cc:1939] T 7e704bfd32e3406da8a1a402f3cfb898 P 08251cc1c3ce4aa29a946868c131a007: Deleting consensus metadata
I20260925 05:10:45.589110 10878 tablet_service.cc:1467] Processing CreateTablet for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a (DEFAULT_TABLE table=test-table [id=23b54b6c8ffb404b9f5eb6a48d4fc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.589242 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.589781 10587 catalog_manager.cc:4841] TS 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859): tablet 7e704bfd32e3406da8a1a402f3cfb898 (table test-table [id=ece5e04131a54de98239a0c237605b46]) successfully deleted
I20260925 05:10:45.590627 11112 tablet_bootstrap.cc:492] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: Bootstrap starting.
I20260925 05:10:45.590834 10586 catalog_manager.cc:4841] TS 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477): tablet 7e704bfd32e3406da8a1a402f3cfb898 (table test-table [id=ece5e04131a54de98239a0c237605b46]) successfully deleted
W20260925 05:10:45.590966 10586 catalog_manager.cc:4808] TS 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477): delete failed for tablet 7e704bfd32e3406da8a1a402f3cfb898 because the tablet was not found. No further retry: Not found: Tablet not found: 7e704bfd32e3406da8a1a402f3cfb898
I20260925 05:10:45.591022 11112 tablet_bootstrap.cc:654] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.591190 10737 tablet_service.cc:1467] Processing CreateTablet for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a (DEFAULT_TABLE table=test-table [id=23b54b6c8ffb404b9f5eb6a48d4fc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:10:45.591367 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:45.591719 11112 tablet_bootstrap.cc:492] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: No bootstrap required, opened a new log
I20260925 05:10:45.591794 11112 ts_tablet_manager.cc:1397] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:45.591914 11112 raft_consensus.cc:348] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.592017 11112 raft_consensus.cc:374] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.592046 11112 raft_consensus.cc:729] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e262f69c3b43fbb0a585880ab04b23, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.592090 11112 consensus_queue.cc:260] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.592181 11112 ts_tablet_manager.cc:1428] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:45.592794 11114 tablet_bootstrap.cc:492] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007: Bootstrap starting.
I20260925 05:10:45.593175 11114 tablet_bootstrap.cc:654] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:45.593672 11113 tablet_bootstrap.cc:492] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: No bootstrap required, opened a new log
I20260925 05:10:45.593734 11113 ts_tablet_manager.cc:1397] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 05:10:45.593834 11113 raft_consensus.cc:348] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.593909 11113 raft_consensus.cc:374] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.593930 11113 raft_consensus.cc:729] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324c9dfbf6b44f0bad0a2db19199450, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.593971 11113 consensus_queue.cc:260] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.594066 11113 ts_tablet_manager.cc:1428] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:45.595216 11135 ts_tablet_manager.cc:1905] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.596748 11135 ts_tablet_manager.cc:1918] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:10:45.596809 11135 log.cc:1196] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/7e704bfd32e3406da8a1a402f3cfb898
I20260925 05:10:45.597023 11135 ts_tablet_manager.cc:1939] T 7e704bfd32e3406da8a1a402f3cfb898 P c0e262f69c3b43fbb0a585880ab04b23: Deleting consensus metadata
I20260925 05:10:45.597383 10588 catalog_manager.cc:4841] TS c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163): tablet 7e704bfd32e3406da8a1a402f3cfb898 (table test-table [id=ece5e04131a54de98239a0c237605b46]) successfully deleted
I20260925 05:10:45.600340 11114 tablet_bootstrap.cc:492] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007: No bootstrap required, opened a new log
I20260925 05:10:45.600411 11114 ts_tablet_manager.cc:1397] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260925 05:10:45.600569 11114 raft_consensus.cc:348] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.600654 11114 raft_consensus.cc:374] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:45.600675 11114 raft_consensus.cc:729] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08251cc1c3ce4aa29a946868c131a007, State: Initialized, Role: FOLLOWER
I20260925 05:10:45.600723 11114 consensus_queue.cc:260] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.600812 11114 ts_tablet_manager.cc:1428] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 05:10:45.770118 11119 raft_consensus.cc:482] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:10:45.770228 11119 raft_consensus.cc:504] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.770433 11119 leader_election.cc:288] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477), 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859)
I20260925 05:10:45.770709 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1dd4e3ceba4098b61bd9e335a3ca0a" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007" is_pre_election: true
I20260925 05:10:45.770720 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1dd4e3ceba4098b61bd9e335a3ca0a" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324c9dfbf6b44f0bad0a2db19199450" is_pre_election: true
I20260925 05:10:45.770828 10758 raft_consensus.cc:2454] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 0.
I20260925 05:10:45.770826 11038 raft_consensus.cc:2454] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 0.
I20260925 05:10:45.770996 10831 leader_election.cc:302] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8324c9dfbf6b44f0bad0a2db19199450, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.771138 11119 raft_consensus.cc:2781] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:10:45.771181 11119 raft_consensus.cc:482] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:10:45.771205 11119 raft_consensus.cc:3037] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.771955 11119 raft_consensus.cc:504] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.772099 11119 leader_election.cc:288] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 election: Requested vote from peers 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477), 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859)
I20260925 05:10:45.772275 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1dd4e3ceba4098b61bd9e335a3ca0a" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08251cc1c3ce4aa29a946868c131a007"
I20260925 05:10:45.772346 10758 raft_consensus.cc:3037] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.772279 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1dd4e3ceba4098b61bd9e335a3ca0a" candidate_uuid: "c0e262f69c3b43fbb0a585880ab04b23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324c9dfbf6b44f0bad0a2db19199450"
I20260925 05:10:45.772377 11038 raft_consensus.cc:3037] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:45.772972 10758 raft_consensus.cc:2454] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 1.
I20260925 05:10:45.773144 11038 raft_consensus.cc:2454] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e262f69c3b43fbb0a585880ab04b23 in term 1.
I20260925 05:10:45.773188 10830 leader_election.cc:302] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08251cc1c3ce4aa29a946868c131a007, c0e262f69c3b43fbb0a585880ab04b23; no voters:
I20260925 05:10:45.773324 11119 raft_consensus.cc:2781] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:45.773435 11119 raft_consensus.cc:686] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 1 LEADER]: Becoming Leader. State: Replica: c0e262f69c3b43fbb0a585880ab04b23, State: Running, Role: LEADER
I20260925 05:10:45.773491 11119 consensus_queue.cc:237] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } }
I20260925 05:10:45.774113 10600 catalog_manager.cc:5495] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130). New cstate: current_term: 1 leader_uuid: "c0e262f69c3b43fbb0a585880ab04b23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e262f69c3b43fbb0a585880ab04b23" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 } health_report { overall_health: UNKNOWN } } }
I20260925 05:10:45.788058 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.785878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.3.6.130:34163)
0925 05:10:45.785930 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.785932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.785940 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:10:45.786004 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.786005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.786013 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.786059 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.786164 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.786644 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.787364 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.787552 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.787635 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.787750 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.787828 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.787881 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.787932 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.787954 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:10:45.793424 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.790839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:58779 (local address 127.3.6.131:40859)
0925 05:10:45.790922 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.790924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.791010 (+ 86us) server_negotiation.cc:407] Connection header received
0925 05:10:45.791125 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.791126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.791145 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.791212 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.791361 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.791909 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.792705 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.792907 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.792986 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.793159 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.793197 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.793259 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.793323 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.793345 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:10:45.793449 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.790723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:40859 (local address 127.3.6.130:58779)
0925 05:10:45.790931 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.790943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.791028 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.791214 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.791215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.791217 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:45.791283 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.791285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.791989 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.791990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.792594 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.792596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.792987 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.792988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.793045 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.793068 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.793262 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.793265 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.793318 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.793370 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260925 05:10:45.795941 11038 raft_consensus.cc:1261] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Refusing update from remote peer c0e262f69c3b43fbb0a585880ab04b23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.796030 10758 raft_consensus.cc:1261] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Refusing update from remote peer c0e262f69c3b43fbb0a585880ab04b23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:10:45.796250 11192 consensus_queue.cc:1035] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8324c9dfbf6b44f0bad0a2db19199450" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 40859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.796399 11119 consensus_queue.cc:1035] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08251cc1c3ce4aa29a946868c131a007" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:10:45.803303 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.800928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:46109 (local address 127.3.6.130:34163)
0925 05:10:45.800970 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:45.800972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:45.800980 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:10:45.801002 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:45.801003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:45.801013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:45.801082 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:10:45.801254 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.801907 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.802651 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.802835 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.802900 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.803018 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.803070 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.803121 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:45.803191 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:10:45.803217 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:10:45.803310 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.800840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:34163 (local address 127.3.6.131:46109)
0925 05:10:45.800881 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.800894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:45.800937 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:45.801096 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:45.801097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:45.801098 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:10:45.801173 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.801180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.801985 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.801985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.802544 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:45.802545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:45.802911 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:45.802912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:45.802952 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:45.802957 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:10:45.803127 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:10:45.803130 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:10:45.803186 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:10:45.803233 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 05:10:45.809940 10600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.810078 10600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:10:45.810722 10600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f1af451a7b34f77b9ace9beb1dcd0ec: Sending DeleteTablet for 3 replicas of tablet ca1dd4e3ceba4098b61bd9e335a3ca0a
I20260925 05:10:45.810992 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:58990
I20260925 05:10:45.811012 11018 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:51040
I20260925 05:10:45.811033 10737 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:10:45 UTC) from {username='oryx', principal='oryx/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45498
I20260925 05:10:45.811097 11135 tablet_replica.cc:330] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: stopping tablet replica
I20260925 05:10:45.811147 11135 raft_consensus.cc:2229] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:10:45.811149 11137 tablet_replica.cc:330] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007: stopping tablet replica
I20260925 05:10:45.811208 11137 raft_consensus.cc:2229] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.811216 11135 raft_consensus.cc:2258] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.811246 11137 raft_consensus.cc:2258] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 08251cc1c3ce4aa29a946868c131a007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.811288 11136 tablet_replica.cc:330] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: stopping tablet replica
I20260925 05:10:45.811339 11136 raft_consensus.cc:2229] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:10:45.811373 11136 raft_consensus.cc:2258] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:10:45.811488 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 10673
I20260925 05:10:45.813903 11135 ts_tablet_manager.cc:1905] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.813910 11136 ts_tablet_manager.cc:1905] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:10:45.815692 11136 ts_tablet_manager.cc:1918] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:45.815721 11135 ts_tablet_manager.cc:1918] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:10:45.815814 11136 log.cc:1196] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/ca1dd4e3ceba4098b61bd9e335a3ca0a
I20260925 05:10:45.815817 11135 log.cc:1196] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/ca1dd4e3ceba4098b61bd9e335a3ca0a
I20260925 05:10:45.816092 11136 ts_tablet_manager.cc:1939] T ca1dd4e3ceba4098b61bd9e335a3ca0a P 8324c9dfbf6b44f0bad0a2db19199450: Deleting consensus metadata
I20260925 05:10:45.816128 11135 ts_tablet_manager.cc:1939] T ca1dd4e3ceba4098b61bd9e335a3ca0a P c0e262f69c3b43fbb0a585880ab04b23: Deleting consensus metadata
I20260925 05:10:45.816504 10587 catalog_manager.cc:4841] TS 8324c9dfbf6b44f0bad0a2db19199450 (127.3.6.131:40859): tablet ca1dd4e3ceba4098b61bd9e335a3ca0a (table test-table [id=23b54b6c8ffb404b9f5eb6a48d4fc418]) successfully deleted
I20260925 05:10:45.816504 10588 catalog_manager.cc:4841] TS c0e262f69c3b43fbb0a585880ab04b23 (127.3.6.130:34163): tablet ca1dd4e3ceba4098b61bd9e335a3ca0a (table test-table [id=23b54b6c8ffb404b9f5eb6a48d4fc418]) successfully deleted
W20260925 05:10:45.820086 10586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:40477 (error 108)
I20260925 05:10:45.820381 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 10813
I20260925 05:10:45.821496 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:45.820552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40477 (local address 127.0.0.1:45514)
0925 05:10:45.821405 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:45.821455 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":46,"threads_started":1}
W20260925 05:10:45.821668 10586 catalog_manager.cc:4584] TS 08251cc1c3ce4aa29a946868c131a007 (127.3.6.129:40477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca1dd4e3ceba4098b61bd9e335a3ca0a: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40477: connect: Connection refused (error 111)
I20260925 05:10:45.827977 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 10953
I20260925 05:10:45.834422 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 10569
2026-09-25T05:10:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2539 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[11205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[11205](info): set up 2 sockets
Sep 25 05:10:45 dist-test-slave-1lvv krb5kdc[11205](info): commencing operation
krb5kdc: starting...
W20260925 05:10:46.888819 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:46 dist-test-slave-1lvv krb5kdc[11205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:47.023082 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:39433
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:39433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:47.108013 11218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:47.108306 11218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:47.108351 11218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:47.109664 11218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:47.109757 11218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:47.109781 11218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:47.109799 11218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:47.109819 11218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:47.111514 11218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:39433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:47.111850 11218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:47.112119 11218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:47.117388 11223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:47.117388 11226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:47.117573 11218 server_base.cc:1034] running on GCE node
W20260925 05:10:47.117714 11224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:47.118252 11218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:47.118607 11218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:47.119823 11218 hybrid_clock.cc:648] HybridClock initialized: now 1790313047119799 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T05:10:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1285 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:10:47 dist-test-slave-1lvv krb5kdc[11231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:10:47 dist-test-slave-1lvv krb5kdc[11231](info): set up 2 sockets
Sep 25 05:10:47 dist-test-slave-1lvv krb5kdc[11231](info): commencing operation
krb5kdc: starting...
W20260925 05:10:48.168906 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:10:48.304157 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44825
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:48.388959 11244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:48.389262 11244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:48.389307 11244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:48.390589 11244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:48.390676 11244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:48.390702 11244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:48.390723 11244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:48.390753 11244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:48.392442 11244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:48.392779 11244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:48.393045 11244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:48.398418 11252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:48.398443 11250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:48.398684 11244 server_base.cc:1034] running on GCE node
W20260925 05:10:48.398437 11249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:48.399036 11244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:48.399336 11244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:48.400513 11244 hybrid_clock.cc:648] HybridClock initialized: now 1790313048400478 us; error 58 us; skew 500 ppm
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:48.408978 11244 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:48.409606 11244 webserver.cc:460] Webserver started at http://127.3.6.190:33177/ using document root <none> and password file <none>
I20260925 05:10:48.409854 11244 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:48.409900 11244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:48.410001 11244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:48.411114 11244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "2e78ad68a0ce65879b81804e4797d648"
server_key_iv: "45a445e45d0cfcdea2bf05d58172194a"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.411460 11244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "2e78ad68a0ce65879b81804e4797d648"
server_key_iv: "45a445e45d0cfcdea2bf05d58172194a"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.413031 11244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:48.413906 11259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.414178 11244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:48.414263 11244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "2e78ad68a0ce65879b81804e4797d648"
server_key_iv: "45a445e45d0cfcdea2bf05d58172194a"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.414327 11244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:48.433007 11244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:48.435315 11244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:48.435447 11244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:48.440027 11244 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44825
I20260925 05:10:48.440079 11311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44825 every 8 connection(s)
I20260925 05:10:48.440456 11244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:48.441416 11312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:10:48.443820 11312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Bootstrap starting.
I20260925 05:10:48.444429 11312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Neither blocks nor log segments found. Creating new log.
I20260925 05:10:48.444684 11312 log.cc:824] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:48.445381 11312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: No bootstrap required, opened a new log
I20260925 05:10:48.447074 11312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:48.447245 11312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:10:48.447270 11312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19942c0e58ff4ef5b0d5c0a73d878dcf, State: Initialized, Role: FOLLOWER
I20260925 05:10:48.447417 11312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:48.447500 11312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:48.447525 11312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:48.447554 11312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:10:48.448168 11312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:48.448261 11312 leader_election.cc:302] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19942c0e58ff4ef5b0d5c0a73d878dcf; no voters:
I20260925 05:10:48.448396 11312 leader_election.cc:288] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:10:48.448500 11315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:10:48.448712 11312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:48.448966 11315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 LEADER]: Becoming Leader. State: Replica: 19942c0e58ff4ef5b0d5c0a73d878dcf, State: Running, Role: LEADER
I20260925 05:10:48.449097 11315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:48.449535 11317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19942c0e58ff4ef5b0d5c0a73d878dcf. Latest consensus state: current_term: 1 leader_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } } }
I20260925 05:10:48.449535 11316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } } }
I20260925 05:10:48.449661 11317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:48.449671 11316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:48.449918 11324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:48.450397 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 11244
I20260925 05:10:48.450510 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:10:48.450583 11324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:48.450759 3098 external_mini_cluster.cc:1428] Setting key 045287428ae44fadb1abaa646dbdfc62
I20260925 05:10:48.452636 11324 catalog_manager.cc:1347] Generated new cluster ID: 4f5c04122fcf4a2f8a8817525d596293
I20260925 05:10:48.452700 11324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:48.462693 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:48.451304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.3.6.190:44825)
0925 05:10:48.451497 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:48.451509 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:48.451742 (+ 233us) server_negotiation.cc:407] Connection header received
0925 05:10:48.451915 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:48.451922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:48.451982 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:48.452153 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:48.452990 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.453963 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.454807 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.455318 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.455453 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:48.458005 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:48.458011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:48.458011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:48.458026 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:48.460391 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:48.460881 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:48.460884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:48.460885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:48.461019 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:48.461244 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:48.461245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:48.461245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:48.461620 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:48.461802 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:48.462315 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 05:10:48.462483 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260925 05:10:48.470498 11324 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:10:48.471127 11324 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:48.481065 11324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Generated new TSK 0
I20260925 05:10:48.481334 11324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:10:48.524319 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44825
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:48.633826 11338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:48.634091 11338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:48.634120 11338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:48.635397 11338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:48.635452 11338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:48.635480 11338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:48.637054 11338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:48.637307 11338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:48.637511 11338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:48.642899 11343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:48.642894 11344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:48.643004 11346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:48.643213 11338 server_base.cc:1034] running on GCE node
I20260925 05:10:48.643417 11338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:48.643710 11338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:48.644932 11338 hybrid_clock.cc:648] HybridClock initialized: now 1790313048644918 us; error 31 us; skew 500 ppm
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:48.647866 11338 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:10:48.648339 11338 webserver.cc:460] Webserver started at http://127.3.6.129:40951/ using document root <none> and password file <none>
I20260925 05:10:48.648550 11338 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:48.648593 11338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:48.648746 11338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:48.649972 11338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "41d913cfa6534d2d8f3464ae53259028"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "49238df1b1c9ada7c722d3437980d515"
server_key_iv: "f528c77eab6c10a305dce43b3680a79b"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.650329 11338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "41d913cfa6534d2d8f3464ae53259028"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "49238df1b1c9ada7c722d3437980d515"
server_key_iv: "f528c77eab6c10a305dce43b3680a79b"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.652037 11338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:10:48.652935 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.653147 11338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:48.653249 11338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "41d913cfa6534d2d8f3464ae53259028"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "49238df1b1c9ada7c722d3437980d515"
server_key_iv: "f528c77eab6c10a305dce43b3680a79b"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.653334 11338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:48.668289 11338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:48.670969 11338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:48.671105 11338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:48.671451 11338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:48.671891 11338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:48.671933 11338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.671957 11338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:48.672001 11338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.678104 11338 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:42843
I20260925 05:10:48.678191 11466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:42843 every 8 connection(s)
I20260925 05:10:48.678573 11338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:48.681448 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 11338
I20260925 05:10:48.681576 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:10:48.681798 3098 external_mini_cluster.cc:1428] Setting key 6309a7db9be3878ded08f96953aaff3f
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:48.690647 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:48.679455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:41359 (local address 127.3.6.190:44825)
0925 05:10:48.679541 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:48.679543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:48.679788 (+ 245us) server_negotiation.cc:407] Connection header received
0925 05:10:48.680336 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:48.680337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:48.680353 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:48.680418 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:48.681702 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.682478 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.684174 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.684437 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.684529 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:48.687114 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:48.687116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:48.687117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:48.687126 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:48.688991 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:48.689519 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:48.689521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:48.689522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:48.689601 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:48.689878 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:48.689879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:48.689880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:48.690048 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:48.690134 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:48.690421 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 05:10:48.690521 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:10:48.690691 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:48.679421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:41359)
0925 05:10:48.679624 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:48.679664 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:48.680218 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:48.680499 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:48.680506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:48.681156 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:48.681507 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:48.681515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.682618 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.682620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:48.684067 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:48.684071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.684545 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.684547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:48.684701 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:48.685311 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:48.685321 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:48.686987 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:48.689113 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:48.689122 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:48.689124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:48.689434 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:48.689687 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:48.689687 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:48.689688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:48.689806 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:48.690122 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:48.690126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:48.690271 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 05:10:48.690366 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:10:48.690502 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260925 05:10:48.691355 11467 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44825
I20260925 05:10:48.691470 11467 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
I20260925 05:10:48.691694 11467 heartbeater.cc:511] Master 127.3.6.190:44825 requested a full tablet report, sending...
I20260925 05:10:48.692335 11276 ts_manager.cc:194] Registered new tserver with Master: 41d913cfa6534d2d8f3464ae53259028 (127.3.6.129:42843)
I20260925 05:10:48.692988 11276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:41359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:10:48.711688 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44825
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:48.796845 11472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:48.797149 11472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:48.797179 11472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:48.798414 11472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:48.798465 11472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:48.798492 11472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:48.800061 11472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:48.800285 11472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:48.800480 11472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:48.806020 11477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:48.806079 11478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:48.805994 11480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:48.806214 11472 server_base.cc:1034] running on GCE node
I20260925 05:10:48.806481 11472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:48.806813 11472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:48.807994 11472 hybrid_clock.cc:648] HybridClock initialized: now 1790313048807970 us; error 47 us; skew 500 ppm
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:48.810734 11472 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:10:48.811172 11472 webserver.cc:460] Webserver started at http://127.3.6.130:37341/ using document root <none> and password file <none>
I20260925 05:10:48.811370 11472 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:48.811410 11472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:48.811508 11472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:48.812659 11472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "e0e7f1e3869542909b6b9c27c5624e0d"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "6106c8dbde955f4ccddfc33fab1efc19"
server_key_iv: "f7268faab470c82ffdf70c95f6b4d589"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.812979 11472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "e0e7f1e3869542909b6b9c27c5624e0d"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "6106c8dbde955f4ccddfc33fab1efc19"
server_key_iv: "f7268faab470c82ffdf70c95f6b4d589"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.814352 11472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:10:48.815112 11487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.815303 11472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:48.815390 11472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "e0e7f1e3869542909b6b9c27c5624e0d"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "6106c8dbde955f4ccddfc33fab1efc19"
server_key_iv: "f7268faab470c82ffdf70c95f6b4d589"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.815449 11472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:48.825066 11472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:48.827291 11472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:48.827483 11472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:48.827708 11472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:48.828099 11472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:48.828135 11472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.828159 11472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:48.828194 11472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.834349 11472 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:37809
I20260925 05:10:48.834427 11600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:37809 every 8 connection(s)
I20260925 05:10:48.834797 11472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:10:48.837647 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 11472
I20260925 05:10:48.837760 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:10:48.837978 3098 external_mini_cluster.cc:1428] Setting key 4b2ce2f1f4bf7566e7f5e9158134d633
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:48.845821 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:48.835640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:45045 (local address 127.3.6.190:44825)
0925 05:10:48.835718 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:48.835721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:48.835984 (+ 263us) server_negotiation.cc:407] Connection header received
0925 05:10:48.836480 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:48.836481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:48.836492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:48.836548 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:48.837615 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.838278 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.839444 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.839810 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.839897 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:48.842538 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:48.842539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:48.842540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:48.842547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:48.844160 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:48.844612 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:48.844613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:48.844614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:48.844678 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:48.844943 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:48.844944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:48.844944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:48.845096 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:48.845171 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:48.845548 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 05:10:48.845674 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:10:48.845927 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:48.835602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:45045)
0925 05:10:48.835869 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:48.835911 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:48.836392 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:48.836607 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:48.836610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:48.837274 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:48.837485 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:48.837490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.838429 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.838430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:48.839352 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:48.839355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.839902 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.839903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:48.840015 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:48.840631 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:48.840644 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:48.842430 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:48.844264 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:48.844273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:48.844275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:48.844520 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:48.844754 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:48.844755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:48.844755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:48.844874 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:48.845154 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:48.845192 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:48.845383 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 05:10:48.845499 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 05:10:48.845687 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
I20260925 05:10:48.846481 11601 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44825
I20260925 05:10:48.846606 11601 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:48.846829 11601 heartbeater.cc:511] Master 127.3.6.190:44825 requested a full tablet report, sending...
I20260925 05:10:48.847252 11276 ts_manager.cc:194] Registered new tserver with Master: e0e7f1e3869542909b6b9c27c5624e0d (127.3.6.130:37809)
I20260925 05:10:48.847821 11276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:45045
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:10:48.867883 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44825
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:48.953702 11606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:48.953984 11606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:48.954031 11606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:48.955389 11606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:48.955475 11606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:48.955526 11606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:48.957420 11606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:48.957741 11606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:48.958005 11606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:48.963938 11612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:48.963989 11614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:48.964002 11611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:48.964102 11606 server_base.cc:1034] running on GCE node
I20260925 05:10:48.964469 11606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:48.964793 11606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:48.965957 11606 hybrid_clock.cc:648] HybridClock initialized: now 1790313048965935 us; error 38 us; skew 500 ppm
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:48.968840 11606 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:10:48.969425 11606 webserver.cc:460] Webserver started at http://127.3.6.131:44707/ using document root <none> and password file <none>
I20260925 05:10:48.969695 11606 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:48.969779 11606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:48.969940 11606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:10:48.971107 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "50ff65d80e6a4442bbe89201884a4fde"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "5246b769e3ed77f876051a867c6ec75f"
server_key_iv: "060119b345c5c7cecca8491baa8c3bc5"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.971575 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "50ff65d80e6a4442bbe89201884a4fde"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "5246b769e3ed77f876051a867c6ec75f"
server_key_iv: "060119b345c5c7cecca8491baa8c3bc5"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.973234 11606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:48.974181 11621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.974421 11606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:48.974536 11606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "50ff65d80e6a4442bbe89201884a4fde"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "5246b769e3ed77f876051a867c6ec75f"
server_key_iv: "060119b345c5c7cecca8491baa8c3bc5"
server_key_version: "encryptionkey@0"
I20260925 05:10:48.974622 11606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:48.985968 11606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:48.988215 11606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:48.988412 11606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:48.988687 11606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:48.989043 11606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:48.989100 11606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.989147 11606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:48.989171 11606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:48.995599 11606 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:33985
I20260925 05:10:48.995641 11734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:33985 every 8 connection(s)
I20260925 05:10:48.996062 11606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:10:48 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313048, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:10:49.003646 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 11606
I20260925 05:10:49.003808 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:10:49.004001 3098 external_mini_cluster.cc:1428] Setting key 786c9d43c9c75dd25c2f30ac5644ed75
I20260925 05:10:49.006668 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:48.996928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:60363 (local address 127.3.6.190:44825)
0925 05:10:48.996988 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:48.996990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:48.997238 (+ 248us) server_negotiation.cc:407] Connection header received
0925 05:10:48.997781 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:48.997782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:48.997795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:48.997856 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:48.998897 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.999410 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.000544 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.000756 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.000855 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.003199 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:49.003201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:49.003201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:49.003209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:49.005115 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:49.005570 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:49.005571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:49.005572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:49.005640 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:49.005915 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:49.005916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:49.005917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:49.006064 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:49.006145 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:49.006468 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 05:10:49.006568 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:10:49.006893 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:48.996916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:60363)
0925 05:10:48.997102 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:48.997143 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:48.997672 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:48.997935 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:48.997940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:48.998566 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:48.998778 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:48.998783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:48.999535 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:48.999536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.000444 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.000446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.000881 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.000882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.001011 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.001556 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.001566 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.003080 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:10:49.005197 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:49.005207 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:49.005210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:49.005476 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:49.005709 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:10:49.005710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:10:49.005710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:10:49.005842 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:10:49.006135 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:10:49.006143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:10:49.006302 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 05:10:49.006427 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 05:10:49.006632 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260925 05:10:49.007424 11735 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44825
I20260925 05:10:49.007515 11735 heartbeater.cc:464] Registering TS with master...
I20260925 05:10:49.007678 11735 heartbeater.cc:511] Master 127.3.6.190:44825 requested a full tablet report, sending...
I20260925 05:10:49.008113 11276 ts_manager.cc:194] Registered new tserver with Master: 50ff65d80e6a4442bbe89201884a4fde (127.3.6.131:33985)
I20260925 05:10:49.008525 11276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:60363
I20260925 05:10:49.016033 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab.
Entry for principal oryx/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab.
WARNING: no policy specified for oryx/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab.
Entry for principal oryx/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab.
WARNING: no policy specified for oryx/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab.
Entry for principal oryx/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab.
I20260925 05:10:49.103739 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 11338
I20260925 05:10:49.108866 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 11472
I20260925 05:10:49.113517 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 11606
I20260925 05:10:49.118160 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 11244
I20260925 05:10:49.124146 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44825
--webserver_interface=127.3.6.190
--webserver_port=33177
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:10:49.207458 11744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:49.207758 11744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:49.207835 11744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:49.209110 11744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:10:49.209187 11744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:49.209211 11744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:49.209234 11744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:10:49.209262 11744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:10:49.210835 11744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:49.211138 11744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:49.211388 11744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:49.216605 11750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:49.216588 11749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:49.216647 11744 server_base.cc:1034] running on GCE node
W20260925 05:10:49.216588 11752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:49.217242 11744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:49.217554 11744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:49.218725 11744 hybrid_clock.cc:648] HybridClock initialized: now 1790313049218699 us; error 40 us; skew 500 ppm
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313049, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:49.221601 11744 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:10:49.222096 11744 webserver.cc:460] Webserver started at http://127.3.6.190:33177/ using document root <none> and password file <none>
I20260925 05:10:49.222353 11744 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:49.222451 11744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:49.224042 11744 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:10:49.224824 11759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.225018 11744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:49.225138 11744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "2e78ad68a0ce65879b81804e4797d648"
server_key_iv: "45a445e45d0cfcdea2bf05d58172194a"
server_key_version: "encryptionkey@0"
I20260925 05:10:49.225528 11744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:49.235241 11744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:49.237648 11744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:49.237831 11744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:49.242022 11744 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44825
I20260925 05:10:49.242069 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44825 every 8 connection(s)
I20260925 05:10:49.242507 11744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:10:49.244118 11812 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:10:49.244746 11812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Bootstrap starting.
I20260925 05:10:49.249647 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 11744
I20260925 05:10:49.249957 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:42843
--local_ip_for_outbound_sockets=127.3.6.129
--tserver_master_addrs=127.3.6.190:44825
--webserver_port=40951
--webserver_interface=127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:10:49.254165 11812 log.cc:824] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Log is configured to *not* fsync() on all Append() calls
I20260925 05:10:49.256093 11812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:10:49.256399 11812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Bootstrap complete.
I20260925 05:10:49.258810 11812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:49.259078 11812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19942c0e58ff4ef5b0d5c0a73d878dcf, State: Initialized, Role: FOLLOWER
I20260925 05:10:49.259286 11812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:49.259383 11812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:10:49.259439 11812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:10:49.259490 11812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:10:49.260632 11812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:49.260756 11812 leader_election.cc:302] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19942c0e58ff4ef5b0d5c0a73d878dcf; no voters:
I20260925 05:10:49.260982 11812 leader_election.cc:288] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:10:49.261020 11816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:10:49.261144 11816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [term 2 LEADER]: Becoming Leader. State: Replica: 19942c0e58ff4ef5b0d5c0a73d878dcf, State: Running, Role: LEADER
I20260925 05:10:49.261237 11816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } }
I20260925 05:10:49.261289 11812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:10:49.261538 11817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } } }
I20260925 05:10:49.261636 11817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:49.261807 11818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19942c0e58ff4ef5b0d5c0a73d878dcf. Latest consensus state: current_term: 2 leader_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19942c0e58ff4ef5b0d5c0a73d878dcf" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44825 } } }
I20260925 05:10:49.261863 11818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf [sys.catalog]: This master's current role is: LEADER
I20260925 05:10:49.261940 11820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:10:49.262570 11820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:10:49.262785 11820 catalog_manager.cc:1259] Loaded cluster ID: 4f5c04122fcf4a2f8a8817525d596293
I20260925 05:10:49.263022 11820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:10:49.263945 11820 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:10:49.264140 11820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19942c0e58ff4ef5b0d5c0a73d878dcf: Loaded TSK: 0
I20260925 05:10:49.264382 11820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:10:49.350049 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:49.350338 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:49.350385 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:49.351691 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:49.351809 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:49.351873 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:10:49.353492 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.129.keytab
--principal=oryx/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:42843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=40951
--tserver_master_addrs=127.3.6.190:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:49.353817 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:49.354070 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:49.359592 11838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:49.359607 11841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:49.359704 11839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:49.359822 11814 server_base.cc:1034] running on GCE node
I20260925 05:10:49.360033 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:49.360353 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:49.361513 11814 hybrid_clock.cc:648] HybridClock initialized: now 1790313049361497 us; error 35 us; skew 500 ppm
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313049, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:49.364675 11814 init.cc:377] Logged in from keytab as oryx/127.3.6.129@KRBTEST.COM (short username oryx)
I20260925 05:10:49.365322 11814 webserver.cc:460] Webserver started at http://127.3.6.129:40951/ using document root <none> and password file <none>
I20260925 05:10:49.365595 11814 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:49.365692 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:49.367704 11814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:10:49.368619 11848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.368934 11814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:49.369057 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "41d913cfa6534d2d8f3464ae53259028"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "49238df1b1c9ada7c722d3437980d515"
server_key_iv: "f528c77eab6c10a305dce43b3680a79b"
server_key_version: "encryptionkey@0"
I20260925 05:10:49.369652 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:49.387739 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:49.390563 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:49.390759 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:49.391067 11814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:49.391474 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:49.391525 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.391558 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:49.391577 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.399441 11814 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:42843
I20260925 05:10:49.399480 11961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:42843 every 8 connection(s)
I20260925 05:10:49.400035 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:10:49.406818 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 11814
I20260925 05:10:49.407229 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:37809
--local_ip_for_outbound_sockets=127.3.6.130
--tserver_master_addrs=127.3.6.190:44825
--webserver_port=37341
--webserver_interface=127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:49.410955 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.400940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:35869)
0925 05:10:49.401173 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.401249 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.401778 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.402297 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.402300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.402951 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.403168 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.403174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.404994 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.404996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.405887 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.405909 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.406793 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.406794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.406964 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.407705 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.407718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.410732 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
W20260925 05:10:49.411558 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.401085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:35869 (local address 127.3.6.190:44825)
0925 05:10:49.401300 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.401310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.401361 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:10:49.402017 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.402025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.402085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.402249 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.403324 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.404863 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.406014 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.406671 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.406789 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.411366 (+ 4577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:35869: BlockingRecv error: recv got EOF from 127.3.6.129:35869 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
W20260925 05:10:49.411883 11962 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:44825 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.6.190:44825: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:49.418767 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.412214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:58679)
0925 05:10:49.412342 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.412354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.412414 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.412591 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.412592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.412776 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.412851 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.412854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.414079 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.414080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.415017 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.415020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.415541 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.415543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.415626 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.416207 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.416214 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.418683 (+ 2469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260925 05:10:49.418941 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.412292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:58679 (local address 127.3.6.190:44825)
0925 05:10:49.412338 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.412340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.412395 (+ 55us) server_negotiation.cc:407] Connection header received
0925 05:10:49.412487 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.412488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.412501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.412568 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.413139 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.413971 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.415143 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.415455 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.415525 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.418845 (+ 3320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:58679: BlockingRecv error: recv got EOF from 127.3.6.129:58679 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:49.424906 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.419341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:58577)
0925 05:10:49.419439 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.419450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.419490 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.419671 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.419672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.419897 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.419955 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.419958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.421005 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.421007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.421722 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.421726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.422220 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.422221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.422275 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.422800 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.422805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.424847 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 05:10:49.425122 11962 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:10:49.425390 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.419463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:58577 (local address 127.3.6.190:44825)
0925 05:10:49.419508 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.419510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.419542 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:10:49.419573 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.419574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.419587 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.419650 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.420054 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.420895 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.421817 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.422136 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.422204 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.424965 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:58577: BlockingRecv error: recv got EOF from 127.3.6.129:58577 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 05:10:49.498297 11966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:49.498540 11966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:49.498569 11966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:49.499807 11966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:49.499854 11966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:49.499883 11966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:10:49.501382 11966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.130.keytab
--principal=oryx/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:37809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=37341
--tserver_master_addrs=127.3.6.190:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:49.501614 11966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:49.501819 11966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:49.507227 11971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:49.507232 11972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:49.507545 11966 server_base.cc:1034] running on GCE node
W20260925 05:10:49.507247 11974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:49.507910 11966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:49.508184 11966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:49.509385 11966 hybrid_clock.cc:648] HybridClock initialized: now 1790313049509378 us; error 44 us; skew 500 ppm
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313049, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:49.512302 11966 init.cc:377] Logged in from keytab as oryx/127.3.6.130@KRBTEST.COM (short username oryx)
I20260925 05:10:49.512888 11966 webserver.cc:460] Webserver started at http://127.3.6.130:37341/ using document root <none> and password file <none>
I20260925 05:10:49.513145 11966 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:49.513293 11966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:49.514858 11966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:49.515478 11981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.515661 11966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:10:49.515796 11966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "e0e7f1e3869542909b6b9c27c5624e0d"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "6106c8dbde955f4ccddfc33fab1efc19"
server_key_iv: "f7268faab470c82ffdf70c95f6b4d589"
server_key_version: "encryptionkey@0"
I20260925 05:10:49.516193 11966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:49.534013 11966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:49.536891 11966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:49.537102 11966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:49.537370 11966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:49.537719 11966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:49.537776 11966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.537819 11966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:49.537864 11966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.544251 11966 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:37809
I20260925 05:10:49.544292 12094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:37809 every 8 connection(s)
I20260925 05:10:49.544924 11966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:10:49.552899 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.545797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:58953 (local address 127.3.6.190:44825)
0925 05:10:49.545865 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.545868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.546061 (+ 193us) server_negotiation.cc:407] Connection header received
0925 05:10:49.546556 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.546557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.546569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.546635 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.547615 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.548186 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.549258 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.549440 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.549496 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.552836 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:58953: BlockingRecv error: recv got EOF from 127.3.6.130:58953 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:10:49.552978 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.545759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:58953)
0925 05:10:49.545924 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.545969 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.546461 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.546693 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.546696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.547307 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.547525 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.547530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.548274 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.548275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.549165 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.549173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.549548 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.549549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.549653 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.550270 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.550279 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.552747 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
W20260925 05:10:49.553277 12095 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:44825 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.6.190:44825: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
I20260925 05:10:49.554768 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 11966
I20260925 05:10:49.555106 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:33985
--local_ip_for_outbound_sockets=127.3.6.131
--tserver_master_addrs=127.3.6.190:44825
--webserver_port=44707
--webserver_interface=127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:49.558378 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.553551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:55501)
0925 05:10:49.553610 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.553622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.553703 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.553841 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.553842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.553992 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.554049 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.554051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.554753 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.554755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.555432 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.555434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.555811 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.555812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.555883 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.556390 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.556395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.558323 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 05:10:49.558548 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.553643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:55501 (local address 127.3.6.190:44825)
0925 05:10:49.553701 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.553703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.553714 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:49.553767 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.553768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.553776 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.553837 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.554141 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.554657 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.555519 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.555712 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.555815 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.558460 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:55501: BlockingRecv error: recv got EOF from 127.3.6.130:55501 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:49.564287 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.558852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:59403)
0925 05:10:49.558996 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.559007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.559065 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.559211 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.559212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.559335 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.559372 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.559373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.560400 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.560400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.560907 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.560908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.561481 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.561482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.561527 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.562099 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.562106 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.564229 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 05:10:49.564467 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.558919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:59403 (local address 127.3.6.190:44825)
0925 05:10:49.558968 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.558970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.559061 (+ 91us) server_negotiation.cc:407] Connection header received
0925 05:10:49.559130 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.559131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.559142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.559197 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.559446 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.560306 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.561012 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.561401 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.561485 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.564375 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:59403: BlockingRecv error: recv got EOF from 127.3.6.130:59403 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 05:10:49.564539 12095 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:10:49.645417 12098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:10:49.645684 12098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:10:49.645712 12098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:10:49.646953 12098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:10:49.646996 12098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:10:49.647024 12098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:10:49.648645 12098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/krb5kdc/oryx_127.3.6.131.keytab
--principal=oryx/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:33985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=44707
--tserver_master_addrs=127.3.6.190:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:10:49.648877 12098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:10:49.649085 12098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:10:49.654547 12103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:10:49.654559 12104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:49.654637 12098 server_base.cc:1034] running on GCE node
W20260925 05:10:49.654559 12106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:10:49.654944 12098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:10:49.655203 12098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:10:49.656397 12098 hybrid_clock.cc:648] HybridClock initialized: now 1790313049656374 us; error 40 us; skew 500 ppm
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313049, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:10:49.659116 12098 init.cc:377] Logged in from keytab as oryx/127.3.6.131@KRBTEST.COM (short username oryx)
I20260925 05:10:49.659643 12098 webserver.cc:460] Webserver started at http://127.3.6.131:44707/ using document root <none> and password file <none>
I20260925 05:10:49.659914 12098 fs_manager.cc:359] Metadata directory not provided
I20260925 05:10:49.659978 12098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:10:49.661515 12098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:49.662128 12113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.662374 12098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:10:49.662478 12098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "50ff65d80e6a4442bbe89201884a4fde"
format_stamp: "Formatted at 2026-09-25 05:10:48 on dist-test-slave-1lvv"
server_key: "5246b769e3ed77f876051a867c6ec75f"
server_key_iv: "060119b345c5c7cecca8491baa8c3bc5"
server_key_version: "encryptionkey@0"
I20260925 05:10:49.662897 12098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:10:49.673200 12098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:10:49.675861 12098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:10:49.676038 12098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:10:49.676307 12098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:10:49.676640 12098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:10:49.676707 12098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.676750 12098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:10:49.676780 12098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:10:49.682988 12098 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:33985
I20260925 05:10:49.683032 12226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:33985 every 8 connection(s)
I20260925 05:10:49.683439 12098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:10:49.691174 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.684308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:33271 (local address 127.3.6.190:44825)
0925 05:10:49.684399 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.684403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.684614 (+ 211us) server_negotiation.cc:407] Connection header received
0925 05:10:49.685115 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.685117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.685127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.685202 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.686212 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.686708 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.687847 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.688034 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.688092 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.691109 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:33271: BlockingRecv error: recv got EOF from 127.3.6.131:33271 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:10:49.691190 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.684277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:33271)
0925 05:10:49.684476 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.684517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.685015 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.685268 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.685272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.685884 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.686117 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.686123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.686801 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.686802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.687687 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.687689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.688122 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.688123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.688246 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.688794 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.688802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.691012 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
W20260925 05:10:49.691473 12227 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:44825 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.6.190:44825: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
I20260925 05:10:49.692093 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 12098
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:49.696544 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.691729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:43273)
0925 05:10:49.691801 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.691813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.691862 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.692148 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.692149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.692292 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.692340 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.692342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.693156 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.693157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.693677 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.693679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.694056 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.694057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.694126 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.694611 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.694615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.696462 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:10:49.696694 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.691982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:43273 (local address 127.3.6.190:44825)
0925 05:10:49.692025 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.692026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.692033 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:10:49.692054 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.692066 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.692074 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.692142 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.692430 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.693084 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.693779 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.693986 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.694066 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.696589 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:43273: BlockingRecv error: recv got EOF from 127.3.6.131:43273 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:10:49.700734 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.692495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.3.6.190:44825)
0925 05:10:49.692639 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.692641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.692787 (+ 146us) server_negotiation.cc:407] Connection header received
0925 05:10:49.692818 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.692819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.692830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.692902 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.693473 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.693964 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.694939 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.695160 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.695240 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.696782 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:10:49.696788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:10:49.696789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:10:49.696806 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:10:49.698924 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:49.699397 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:49.699401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:49.699401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:49.699515 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:10:49.699812 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:10:49.699813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:10:49.699814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:10:49.700195 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:10:49.700353 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:10:49.700462 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 05:10:49.700604 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:49 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:49.702246 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:49.696985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:37291)
0925 05:10:49.697100 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:49.697113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:49.697155 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:49.697307 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:49.697308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:49.697449 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:49.697494 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.697497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.698329 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.698330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.699124 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:49.699127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.699567 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.699568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:49.699630 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.700045 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:49.700051 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:49.702193 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 05:10:49.702386 12227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:10:49.702432 11965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:49.697048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:37291 (local address 127.3.6.190:44825)
0925 05:10:49.697124 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:49.697127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:49.697139 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:49.697223 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:49.697224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:49.697234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:49.697296 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:49.697607 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.698244 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.699206 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:49.699493 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:49.699546 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:49.702369 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:37291: BlockingRecv error: recv got EOF from 127.3.6.131:37291 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 05:10:50 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:50 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:50.430748 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:50.425602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:44409)
0925 05:10:50.425787 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:50.425801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:50.425870 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:50.426020 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:50.426021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:50.426195 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:50.426247 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:50.426249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.426922 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.426923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:50.427494 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:50.427496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.427944 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.427945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:50.428012 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:50.428563 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:50.428568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:50.430636 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
W20260925 05:10:50.431030 12233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:50.425717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:44409 (local address 127.3.6.190:44825)
0925 05:10:50.425859 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:50.425861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:50.425873 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:50.425932 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:50.425933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:50.425943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:50.425995 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:50.426319 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.426823 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.427607 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.427824 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.427920 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:50.430718 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:44409: BlockingRecv error: recv got EOF from 127.3.6.129:44409 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
Sep 25 05:10:50 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:50 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:50.570191 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:50.565041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:48241)
0925 05:10:50.565214 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:50.565226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:50.565292 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:50.565492 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:50.565492 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:50.565684 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:50.565736 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:50.565737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.566394 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.566394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:50.567030 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:50.567033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.567443 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.567443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:50.567509 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:50.568002 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:50.568007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:50.570081 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
W20260925 05:10:50.570261 12233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:50.565113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:48241 (local address 127.3.6.190:44825)
0925 05:10:50.565230 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:50.565232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:50.565310 (+ 78us) server_negotiation.cc:407] Connection header received
0925 05:10:50.565350 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:50.565351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:50.565362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:50.565455 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:50.565801 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.566311 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.567143 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.567357 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.567433 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:50.570195 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:48241: BlockingRecv error: recv got EOF from 127.3.6.130:48241 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 05:10:50 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:50 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:50.707975 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:50.702934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:41853)
0925 05:10:50.703133 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:50.703145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:50.703209 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:50.703357 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:50.703357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:50.703557 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:50.703607 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:50.703609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.704352 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.704353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:50.704924 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:50.704926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.705293 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.705293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:50.705332 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:50.705865 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:50.705870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:50.707899 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":130,"threads_started":1}
W20260925 05:10:50.708103 12233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:50.703042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:41853 (local address 127.3.6.190:44825)
0925 05:10:50.703124 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:50.703127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:50.703219 (+ 92us) server_negotiation.cc:407] Connection header received
0925 05:10:50.703254 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:50.703255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:50.703267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:50.703334 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:50.703694 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.704271 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.705014 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:50.705200 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:50.705293 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:50.708000 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:41853: BlockingRecv error: recv got EOF from 127.3.6.131:41853 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 05:10:51 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:51 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:51.436321 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:51.431307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:54229)
0925 05:10:51.431494 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:51.431506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:51.431569 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:51.431720 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:51.431720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:51.431908 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:51.431961 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:51.431963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.432623 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.432624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:51.433158 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:51.433160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.433600 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.433601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:51.433671 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:51.434173 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:51.434178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:51.436222 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
W20260925 05:10:51.436408 12237 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:51.431422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:54229 (local address 127.3.6.190:44825)
0925 05:10:51.431554 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:51.431555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:51.431566 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:10:51.431622 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:51.431623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:51.431634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:51.431696 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:51.432030 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.432544 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.433283 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.433512 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.433601 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:51.436330 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:54229: BlockingRecv error: recv got EOF from 127.3.6.129:54229 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
Sep 25 05:10:51 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:51 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:51.576265 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:51.570834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:56659)
0925 05:10:51.571020 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:51.571045 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:51.571124 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:51.571273 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:51.571274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:51.571459 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:51.571511 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:51.571512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.572373 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.572375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:51.572915 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:51.572916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.573330 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.573331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:51.573394 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:51.573971 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:51.573976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:51.576197 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
W20260925 05:10:51.576371 12237 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:51.570976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:56659 (local address 127.3.6.190:44825)
0925 05:10:51.571050 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:51.571052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:51.571155 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:10:51.571184 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:51.571186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:51.571197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:51.571253 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:51.571592 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.572266 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.573024 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.573235 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.573303 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:51.576300 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:56659: BlockingRecv error: recv got EOF from 127.3.6.130:56659 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 05:10:51 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:51 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:51.713274 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:51.708485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:46209)
0925 05:10:51.708774 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:51.708789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:51.708852 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:51.708989 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:51.708989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:51.709170 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:51.709221 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:51.709223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.709858 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.709858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:51.710396 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:51.710398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.710754 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.710755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:51.710819 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:51.711310 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:51.711315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:51.713171 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
W20260925 05:10:51.713330 12237 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:51.708654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:46209 (local address 127.3.6.190:44825)
0925 05:10:51.708718 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:51.708721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:51.708852 (+ 131us) server_negotiation.cc:407] Connection header received
0925 05:10:51.708901 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:51.708902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:51.708914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:51.708979 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:51.709283 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.709788 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.710470 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:51.710660 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:51.710718 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:51.713259 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:46209: BlockingRecv error: recv got EOF from 127.3.6.131:46209 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 05:10:52 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:52 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:52.442656 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:52.436938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:44173)
0925 05:10:52.437115 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:52.437126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:52.437220 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:52.437446 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:52.437447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:52.437718 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:52.437786 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:52.437789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.438540 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.438541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:52.439105 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:52.439107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.439482 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.439483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:52.439560 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:52.440330 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:52.440335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:52.442576 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
W20260925 05:10:52.442919 12241 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:52.437004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:44173 (local address 127.3.6.190:44825)
0925 05:10:52.437179 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:52.437181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:52.437195 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:10:52.437316 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:52.437318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:52.437333 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:52.437423 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:52.437879 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.438425 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.439197 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.439390 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.439448 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:52.442724 (+ 3276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:44173: BlockingRecv error: recv got EOF from 127.3.6.129:44173 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
Sep 25 05:10:52 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:52 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:52.581429 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:52.576835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:60115)
0925 05:10:52.576991 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:52.577001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:52.577063 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:52.577225 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:52.577226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:52.577397 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:52.577447 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:52.577448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.578106 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.578107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:52.578646 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:52.578648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.578988 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.578989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:52.579028 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:52.579561 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:52.579565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:52.581352 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260925 05:10:52.581554 12241 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:52.576909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:60115 (local address 127.3.6.190:44825)
0925 05:10:52.576989 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:52.576990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:52.577067 (+ 77us) server_negotiation.cc:407] Connection header received
0925 05:10:52.577128 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:52.577129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:52.577139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:52.577204 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:52.577514 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.578024 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.578724 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.578920 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.578974 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:52.581458 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:60115: BlockingRecv error: recv got EOF from 127.3.6.130:60115 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 05:10:52 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:52 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:52.719089 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:52.713931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:60287)
0925 05:10:52.714071 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:52.714092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:52.714158 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:52.714389 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:52.714390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:52.714588 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:52.714637 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:52.714639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.715388 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.715389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:52.716181 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:52.716183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.716546 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.716547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:52.716601 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:52.717114 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:52.717118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:52.719002 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
W20260925 05:10:52.719251 12241 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:52.714179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:60287 (local address 127.3.6.190:44825)
0925 05:10:52.714265 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:52.714267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:52.714275 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:10:52.714300 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:52.714301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:52.714313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:52.714382 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:52.714731 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.715307 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.716263 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:52.716461 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:52.716537 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:52.719165 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:60287: BlockingRecv error: recv got EOF from 127.3.6.131:60287 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 05:10:53 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:53 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:53.453759 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:53.443496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:34283)
0925 05:10:53.443678 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:53.443698 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:53.443753 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:53.444024 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:53.444025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:53.444203 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:53.444257 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:53.444258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.444915 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.444915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:53.445441 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:53.445443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.445861 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.445862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:53.445932 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:53.451525 (+ 5593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:53.451532 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:53.453694 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":122,"threads_started":1}
W20260925 05:10:53.453928 12245 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:53.443621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:34283 (local address 127.3.6.190:44825)
0925 05:10:53.443812 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:53.443815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:53.443845 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:10:53.443883 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:53.443884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:53.443898 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:53.443983 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:53.444331 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.444837 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.445530 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.445785 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.445864 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:53.453828 (+ 7964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:34283: BlockingRecv error: recv got EOF from 127.3.6.129:34283 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 25 05:10:53 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:53 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:53.587153 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:53.582092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:41493)
0925 05:10:53.582248 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:53.582263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:53.582340 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:53.582508 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:53.582508 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:53.582707 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:53.582757 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:53.582759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.583455 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.583457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:53.584037 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:53.584039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.584434 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.584435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:53.584492 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:53.585042 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:53.585046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:53.587063 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
W20260925 05:10:53.587278 12245 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:53.582224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:41493 (local address 127.3.6.190:44825)
0925 05:10:53.582290 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:53.582292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:53.582320 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:10:53.582401 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:53.582402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:53.582414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:53.582486 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:53.582846 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.583355 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.584132 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.584338 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.584437 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:53.587210 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:41493: BlockingRecv error: recv got EOF from 127.3.6.130:41493 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 05:10:53 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:53 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:53.724820 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:53.719684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:54981)
0925 05:10:53.719910 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:53.719926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:53.720038 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:53.720220 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:53.720221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:53.720489 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:53.720552 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:53.720555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.721346 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.721347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:53.721909 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:53.721911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.722346 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.722347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:53.722407 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:53.722898 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:53.722902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:53.724759 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":149,"threads_started":1}
W20260925 05:10:53.724975 12245 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:53.719841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:54981 (local address 127.3.6.190:44825)
0925 05:10:53.719915 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:53.719917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:53.720016 (+ 99us) server_negotiation.cc:407] Connection header received
0925 05:10:53.720118 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:53.720119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:53.720141 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:53.720210 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:53.720658 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.721252 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.722017 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:53.722220 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:53.722292 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:53.724891 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:54981: BlockingRecv error: recv got EOF from 127.3.6.131:54981 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 05:10:54 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:54 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:54.459245 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:54.454314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:44795)
0925 05:10:54.454455 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:54.454474 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:54.454528 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:54.454729 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:54.454729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:54.454899 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:54.454952 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:54.454954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.455647 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.455648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:54.456272 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:54.456275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.456681 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.456682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:54.456724 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:54.457246 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:54.457251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:54.459184 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
W20260925 05:10:54.459407 12249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:54.454412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:44795 (local address 127.3.6.190:44825)
0925 05:10:54.454581 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:54.454583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:54.454592 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:54.454625 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:54.454626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:54.454636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:54.454707 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:54.455024 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.455565 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.456373 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.456605 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.456668 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:54.459304 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:44795: BlockingRecv error: recv got EOF from 127.3.6.129:44795 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
Sep 25 05:10:54 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:54 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:54.592451 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:54.587866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:47529)
0925 05:10:54.587999 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:54.588014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:54.588079 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:54.588217 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:54.588217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:54.588384 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:54.588437 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:54.588439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.589099 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.589100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:54.589630 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:54.589631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.589973 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.589974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:54.590014 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:54.590472 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:54.590477 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:54.592387 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260925 05:10:54.592618 12249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:54.587935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:47529 (local address 127.3.6.190:44825)
0925 05:10:54.588007 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:54.588009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:54.588079 (+ 70us) server_negotiation.cc:407] Connection header received
0925 05:10:54.588123 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:54.588124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:54.588135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:54.588194 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:54.588504 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.589023 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.589720 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.589905 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.589959 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:54.592525 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:47529: BlockingRecv error: recv got EOF from 127.3.6.130:47529 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 05:10:54 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:54 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:54.730876 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:54.725409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:58163)
0925 05:10:54.725603 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:54.725614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:54.725663 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:54.725854 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:54.725854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:54.726023 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:54.726075 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:54.726076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.727099 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.727100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:54.727698 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:54.727701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.728105 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.728106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:54.728198 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:54.728792 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:54.728797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:54.730811 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
W20260925 05:10:54.731062 12249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:54.725545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:58163 (local address 127.3.6.190:44825)
0925 05:10:54.725609 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:54.725611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:54.725718 (+ 107us) server_negotiation.cc:407] Connection header received
0925 05:10:54.725761 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:54.725762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:54.725778 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:54.725839 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:54.726151 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.726972 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.727825 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:54.728032 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:54.728089 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:54.730972 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:58163: BlockingRecv error: recv got EOF from 127.3.6.131:58163 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 05:10:55 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:55 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:55.465193 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:55.459916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:60409)
0925 05:10:55.460090 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:55.460102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:55.460189 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:55.460344 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:55.460345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:55.460537 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:55.460587 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:55.460589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.461338 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.461340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:55.461907 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:55.461908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.462347 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.462348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:55.462415 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:55.462934 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:55.462939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:55.465087 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
W20260925 05:10:55.465265 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:55.459933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:60409 (local address 127.3.6.190:44825)
0925 05:10:55.460093 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:55.460095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:55.460169 (+ 74us) server_negotiation.cc:407] Connection header received
0925 05:10:55.460245 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:55.460246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:55.460259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:55.460339 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:55.460668 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.461246 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.461991 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.462265 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.462342 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:55.465186 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:60409: BlockingRecv error: recv got EOF from 127.3.6.129:60409 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
Sep 25 05:10:55 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:55 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:55.597610 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:55.592951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:57917)
0925 05:10:55.593095 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:55.593106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:55.593177 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:55.593293 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:55.593294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:55.593461 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:55.593510 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:55.593512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.594183 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.594184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:55.594756 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:55.594758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.595126 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.595127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:55.595168 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:55.595712 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:55.595717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:55.597552 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
W20260925 05:10:55.597759 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:55.593008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:57917 (local address 127.3.6.190:44825)
0925 05:10:55.593105 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:55.593108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:55.593183 (+ 75us) server_negotiation.cc:407] Connection header received
0925 05:10:55.593221 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:55.593222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:55.593233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:55.593289 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:55.593575 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.594080 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.594851 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.595043 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.595115 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:55.597674 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:57917: BlockingRecv error: recv got EOF from 127.3.6.130:57917 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 05:10:55 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:55 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:55.736716 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:55.731426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:57019)
0925 05:10:55.731644 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:55.731661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:55.731781 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:55.731957 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:55.731958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:55.732204 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:55.732266 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:55.732269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.733100 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.733106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:55.733665 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:55.733667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.734054 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.734055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:55.734097 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:55.734677 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:55.734682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:55.736646 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
W20260925 05:10:55.736917 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:55.731571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:57019 (local address 127.3.6.190:44825)
0925 05:10:55.731670 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:55.731673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:55.731744 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:10:55.731857 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:55.731858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:55.731879 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:55.731948 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:55.732368 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.732953 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.733754 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:55.733955 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:55.734040 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:55.736821 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:57019: BlockingRecv error: recv got EOF from 127.3.6.131:57019 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 05:10:56 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:56 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:56.471038 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:56.465803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:36153)
0925 05:10:56.465982 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:56.465993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:56.466048 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:56.466264 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:56.466265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:56.466448 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:56.466503 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:56.466505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.467328 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.467330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:56.468012 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:56.468016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.468388 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.468389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:56.468430 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:56.468922 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:56.468927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:56.470933 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
W20260925 05:10:56.471087 12257 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:56.465892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:36153 (local address 127.3.6.190:44825)
0925 05:10:56.466024 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:56.466025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:56.466037 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:56.466125 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:56.466126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:56.466141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:56.466235 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:56.466617 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.467226 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.468105 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.468315 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.468375 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:56.471015 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:36153: BlockingRecv error: recv got EOF from 127.3.6.129:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
Sep 25 05:10:56 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:56 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:56.603416 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:56.598201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:46357)
0925 05:10:56.598427 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:56.598442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:56.598529 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:56.598718 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:56.598719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:56.598909 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:56.598957 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:56.598959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.599684 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.599685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:56.600255 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:56.600257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.600666 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.600667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:56.600728 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:56.601287 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:56.601292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:56.603316 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
W20260925 05:10:56.603471 12257 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:56.598335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:46357 (local address 127.3.6.190:44825)
0925 05:10:56.598432 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:56.598436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:56.598507 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:10:56.598594 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:56.598596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:56.598608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:56.598686 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:56.599052 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.599580 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.600359 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.600565 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.600659 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:56.603406 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:46357: BlockingRecv error: recv got EOF from 127.3.6.130:46357 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 05:10:56 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:56 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:56.742059 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:56.737271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:39481)
0925 05:10:56.737414 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:56.737433 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:56.737488 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:56.737666 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:56.737667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:56.737839 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:56.737888 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:56.737890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.738608 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.738609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:56.739160 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:56.739161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.739515 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.739516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:56.739566 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:56.740117 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:56.740121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:56.741997 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260925 05:10:56.742218 12257 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:56.737361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:39481 (local address 127.3.6.190:44825)
0925 05:10:56.737465 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:56.737467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:56.737482 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:10:56.737564 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:56.737565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:56.737580 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:56.737651 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:56.737992 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.738503 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.739245 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:56.739429 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:56.739483 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:56.742133 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:39481: BlockingRecv error: recv got EOF from 127.3.6.131:39481 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 05:10:57 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:57 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:57.476858 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:57.471678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:53493)
0925 05:10:57.471829 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:57.471842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:57.471903 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:57.472362 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:57.472363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:57.472553 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:57.472604 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:57.472606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.473255 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.473255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:57.473785 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:57.473786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.474190 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.474191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:57.474236 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:57.474707 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:57.474712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:57.476738 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
W20260925 05:10:57.480131 12261 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:57.471850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:53493 (local address 127.3.6.190:44825)
0925 05:10:57.471991 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:57.471994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:57.472231 (+ 237us) server_negotiation.cc:407] Connection header received
0925 05:10:57.472260 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:57.472260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:57.472272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:57.472345 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:57.472678 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.473185 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.473905 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.474122 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.474176 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:57.480055 (+ 5879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:53493: BlockingRecv error: recv got EOF from 127.3.6.129:53493 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
Sep 25 05:10:57 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:57 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:57.608835 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:57.604036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:51953)
0925 05:10:57.604167 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:57.604180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:57.604242 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:57.604355 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:57.604355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:57.604524 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:57.604569 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:57.604570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.605376 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.605377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:57.605917 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:57.605919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.606300 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.606301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:57.606350 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:57.606833 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:57.606837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:57.608733 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
W20260925 05:10:57.608894 12261 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:57.604117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:51953 (local address 127.3.6.190:44825)
0925 05:10:57.604235 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:57.604236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:57.604248 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:57.604297 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:57.604298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:57.604307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:57.604351 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:57.604636 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.605272 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.606024 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.606206 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.606258 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:57.608828 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:51953: BlockingRecv error: recv got EOF from 127.3.6.130:51953 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 25 05:10:57 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:57 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:57.747785 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:57.742639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:37935)
0925 05:10:57.742784 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:57.742821 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:57.742888 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:57.743086 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:57.743087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:57.743305 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:57.743363 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:57.743365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.744233 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.744235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:57.744782 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:57.744784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.745174 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.745175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:57.745250 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:57.745705 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:57.745709 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:57.747688 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
W20260925 05:10:57.747946 12261 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:57.742765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:37935 (local address 127.3.6.190:44825)
0925 05:10:57.742866 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:57.742868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:57.742882 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:10:57.742966 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:57.742967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:57.742980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:57.743055 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:57.743457 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.744144 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.744878 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:57.745095 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:57.745176 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:57.747859 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:37935: BlockingRecv error: recv got EOF from 127.3.6.131:37935 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 05:10:58 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:58 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:58.489491 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:58.478377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:48553)
0925 05:10:58.478509 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:58.478527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:58.478585 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:58.478755 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:58.478755 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:58.478917 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:58.478970 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:58.478971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.479623 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.479624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:58.480223 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:58.480226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.480590 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.480591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:58.480634 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:58.487288 (+ 6654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:58.487295 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:58.489429 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260925 05:10:58.489648 12265 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:58.478453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:48553 (local address 127.3.6.190:44825)
0925 05:10:58.478607 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:58.478609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:58.478619 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:10:58.478656 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:58.478656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:58.478670 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:58.478738 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:58.479035 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.479552 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.480305 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.480516 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.480580 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:58.489549 (+ 8969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:48553: BlockingRecv error: recv got EOF from 127.3.6.129:48553 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
Sep 25 05:10:58 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:58 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:58.614454 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:58.609391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:41915)
0925 05:10:58.609693 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:58.609708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:58.609783 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:58.609948 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:58.609950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:58.610142 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:58.610202 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:58.610204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.610916 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.610917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:58.611472 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:58.611473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.611917 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.611918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:58.611975 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:58.612445 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:58.612450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:58.614390 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":211,"threads_started":1}
W20260925 05:10:58.614681 12265 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:58.609620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:41915 (local address 127.3.6.190:44825)
0925 05:10:58.609701 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:58.609704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:58.609763 (+ 59us) server_negotiation.cc:407] Connection header received
0925 05:10:58.609841 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:58.609842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:58.609855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:58.609925 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:58.610307 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.610827 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.611563 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.611823 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.611916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:58.614620 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:41915: BlockingRecv error: recv got EOF from 127.3.6.130:41915 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 05:10:58 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:58 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:58.753348 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:58.748348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:52689)
0925 05:10:58.748507 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:58.748519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:58.748607 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:58.748771 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:58.748772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:58.748963 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:58.749014 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:58.749016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.749719 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.749720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:58.750267 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:58.750269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.750658 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.750659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:58.750723 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:58.751256 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:58.751261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:58.753290 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
W20260925 05:10:58.753500 12265 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:58.748473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:52689 (local address 127.3.6.190:44825)
0925 05:10:58.748582 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:58.748584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:58.748596 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:10:58.748676 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:58.748677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:58.748696 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:58.748762 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:58.749102 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.749620 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.750375 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:58.750575 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:58.750659 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:58.753408 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:52689: BlockingRecv error: recv got EOF from 127.3.6.131:52689 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 05:10:59 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:59 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:59.495003 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:59.490089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:52765)
0925 05:10:59.490234 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:59.490252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:59.490340 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:59.490545 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:59.490546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:59.490723 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:59.490781 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:59.490782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.491562 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.491563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:59.492131 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:59.492133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.492462 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.492463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:59.492502 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:59.492985 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:59.492990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:59.494930 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260925 05:10:59.495113 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:59.490131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:52765 (local address 127.3.6.190:44825)
0925 05:10:59.490284 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:59.490286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:59.490325 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:10:59.490417 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:59.490418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:59.490439 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:59.490523 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:59.490949 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.491462 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.492207 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.492396 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.492449 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:59.495034 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:52765: BlockingRecv error: recv got EOF from 127.3.6.129:52765 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
Sep 25 05:10:59 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:59 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:59.620303 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:59.615041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:57825)
0925 05:10:59.615232 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:59.615244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:59.615341 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:59.615508 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:59.615509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:59.615704 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:59.615751 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:59.615774 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.616470 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.616472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:59.617008 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:59.617009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.617419 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.617420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:59.617486 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:59.617955 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:59.617960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:59.620243 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
W20260925 05:10:59.620478 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:59.615138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:57825 (local address 127.3.6.190:44825)
0925 05:10:59.615228 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:59.615231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:59.615322 (+ 91us) server_negotiation.cc:407] Connection header received
0925 05:10:59.615429 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:59.615431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:59.615442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:59.615499 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:59.615859 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.616379 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.617107 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.617328 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.617409 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:59.620395 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:57825: BlockingRecv error: recv got EOF from 127.3.6.130:57825 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 05:10:59 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:10:59 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:10:59.759191 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:10:59.754075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:36005)
0925 05:10:59.754307 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:10:59.754321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:10:59.754384 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:10:59.754571 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:10:59.754573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:10:59.754855 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:10:59.754924 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:59.754926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.755616 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.755617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:59.756217 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:10:59.756219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.756582 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.756583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:10:59.756634 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:59.757200 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:10:59.757211 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:10:59.759078 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":126,"threads_started":1}
W20260925 05:10:59.759238 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:10:59.754285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:36005 (local address 127.3.6.190:44825)
0925 05:10:59.754362 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:10:59.754365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:10:59.754374 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:10:59.754460 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:10:59.754461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:10:59.754476 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:10:59.754537 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:10:59.755020 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.755532 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.756292 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:10:59.756475 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:10:59.756532 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:10:59.759169 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:36005: BlockingRecv error: recv got EOF from 127.3.6.131:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 05:11:00 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:00 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:00.500713 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:00.495566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:49713)
0925 05:11:00.495724 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:00.495735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:00.495851 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:00.496171 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:00.496172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:00.496359 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:00.496410 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:00.496412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.497174 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.497175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:00.497729 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:00.497731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.498071 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.498072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:00.498111 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:00.498645 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:00.498650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:00.500613 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
W20260925 05:11:00.500795 12273 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:00.495753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:49713 (local address 127.3.6.190:44825)
0925 05:11:00.496002 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:00.496004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:00.496012 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:11:00.496036 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:00.496037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:00.496048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:00.496135 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:00.496529 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.497062 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.497801 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.497997 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.498058 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:00.500716 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:49713: BlockingRecv error: recv got EOF from 127.3.6.129:49713 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
Sep 25 05:11:00 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:00 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:00.625833 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:00.620811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:52817)
0925 05:11:00.621007 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:00.621020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:00.621108 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:00.621273 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:00.621274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:00.621476 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:00.621529 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:00.621530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.622231 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.622233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:00.622771 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:00.622773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.623151 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.623152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:00.623211 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:00.623694 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:00.623699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:00.625726 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
W20260925 05:11:00.625891 12273 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:00.620915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:52817 (local address 127.3.6.190:44825)
0925 05:11:00.621008 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:00.621010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:00.621088 (+ 78us) server_negotiation.cc:407] Connection header received
0925 05:11:00.621173 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:00.621175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:00.621195 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:00.621262 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:00.621630 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.622140 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.622859 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.623060 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.623151 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:00.625823 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:52817: BlockingRecv error: recv got EOF from 127.3.6.130:52817 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 05:11:00 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:00 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:00.765357 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:00.759856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:43687)
0925 05:11:00.759996 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:00.760009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:00.760096 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:00.760328 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:00.760329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:00.760509 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:00.760577 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:00.760579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.761460 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.761462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:00.762241 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:00.762243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.762652 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.762653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:00.762709 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:00.763277 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:00.763281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:00.765294 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260925 05:11:00.765499 12273 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:00.759953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:43687 (local address 127.3.6.190:44825)
0925 05:11:00.760073 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:00.760075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:00.760087 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:11:00.760226 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:00.760227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:00.760241 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:00.760303 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:00.760679 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.761312 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.762337 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:00.762568 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:00.762653 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:00.765403 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:43687: BlockingRecv error: recv got EOF from 127.3.6.131:43687 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 05:11:01 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:01 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:01.506610 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:01.501357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:50779)
0925 05:11:01.501578 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:01.501591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:01.501677 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:01.501835 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:01.501836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:01.502016 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:01.502067 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:01.502069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.502790 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.502791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:01.503325 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:01.503327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.503715 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.503715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:01.503789 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:01.504385 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:01.504391 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:01.506543 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":109,"threads_started":1}
W20260925 05:11:01.506821 12277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:01.501545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:50779 (local address 127.3.6.190:44825)
0925 05:11:01.501680 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:01.501681 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:01.501714 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:11:01.501741 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:01.501742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:01.501753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:01.501810 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:01.502151 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.502699 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.503422 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.503640 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.503703 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:01.506734 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:50779: BlockingRecv error: recv got EOF from 127.3.6.129:50779 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
Sep 25 05:11:01 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:01 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:01.631474 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:01.626376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:40415)
0925 05:11:01.626599 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:01.626610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:01.626694 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:01.626828 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:01.626829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:01.627028 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:01.627077 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:01.627078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.627810 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.627811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:01.628353 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:01.628355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.628784 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.628785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:01.628846 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:01.629334 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:01.629339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:01.631355 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":143,"threads_started":1}
W20260925 05:11:01.631520 12277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:01.626428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:40415 (local address 127.3.6.190:44825)
0925 05:11:01.626476 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:01.626479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:01.626678 (+ 199us) server_negotiation.cc:407] Connection header received
0925 05:11:01.626755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:01.626756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:01.626770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:01.626823 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:01.627162 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.627684 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.628450 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.628703 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.628785 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:01.631452 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:40415: BlockingRecv error: recv got EOF from 127.3.6.130:40415 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 05:11:01 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:01 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:01.771219 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:01.765880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:43091)
0925 05:11:01.766059 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:01.766072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:01.766154 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:01.766311 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:01.766312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:01.766558 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:01.766663 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:01.766666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.767413 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.767414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:01.768042 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:01.768045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.768429 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.768430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:01.768509 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:01.769010 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:01.769015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:01.771148 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
W20260925 05:11:01.771314 12277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:01.765981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:43091 (local address 127.3.6.190:44825)
0925 05:11:01.766058 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:01.766061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:01.766132 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:11:01.766214 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:01.766215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:01.766238 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:01.766304 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:01.766766 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.767307 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.768147 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:01.768342 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:01.768426 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:01.771246 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:43091: BlockingRecv error: recv got EOF from 127.3.6.131:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 05:11:02 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:02 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:02.512327 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:02.507303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:56937)
0925 05:11:02.507480 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:02.507492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:02.507563 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:02.507709 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:02.507710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:02.507965 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:02.508036 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:02.508039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.508917 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.508918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:02.509463 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:02.509466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.509878 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.509879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:02.509921 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:02.510397 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:02.510402 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:02.512257 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
W20260925 05:11:02.512485 12281 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:02.507427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56937 (local address 127.3.6.190:44825)
0925 05:11:02.507574 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:02.507576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:02.507587 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:11:02.507624 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:02.507625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:02.507636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:02.507695 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:02.508120 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.508844 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.509590 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.509801 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.509868 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:02.512393 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:56937: BlockingRecv error: recv got EOF from 127.3.6.129:56937 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
Sep 25 05:11:02 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:02 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:02.637162 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:02.632239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:43039)
0925 05:11:02.632379 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:02.632398 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:02.632475 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:02.632646 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:02.632647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:02.632842 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:02.632892 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:02.632894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.633649 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.633650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:02.634208 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:02.634210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.634596 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.634597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:02.634656 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:02.635181 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:02.635186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:02.637083 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
W20260925 05:11:02.637292 12281 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:02.632395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:43039 (local address 127.3.6.190:44825)
0925 05:11:02.632485 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:02.632487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:02.632499 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:11:02.632558 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:02.632559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:02.632570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:02.632636 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:02.632988 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.633548 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.634303 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.634513 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.634598 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:02.637207 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:43039: BlockingRecv error: recv got EOF from 127.3.6.130:43039 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 05:11:02 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:02 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:02.777256 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:02.771856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:54407)
0925 05:11:02.772027 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:02.772039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:02.772113 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:02.772250 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:02.772251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:02.772413 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:02.772462 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:02.772464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.773227 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.773228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:02.773773 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:02.773775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.774288 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.774289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:02.774352 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:02.774919 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:02.774924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:02.777132 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
W20260925 05:11:02.777307 12281 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:02.771929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:54407 (local address 127.3.6.190:44825)
0925 05:11:02.772017 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:02.772021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:02.772111 (+ 90us) server_negotiation.cc:407] Connection header received
0925 05:11:02.772163 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:02.772164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:02.772178 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:02.772236 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:02.772583 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.773107 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.773896 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:02.774128 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:02.774217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:02.777233 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:54407: BlockingRecv error: recv got EOF from 127.3.6.131:54407 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 05:11:03 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:03 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:03.522501 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:03.512849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:53983)
0925 05:11:03.512998 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:03.513010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:03.513076 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:03.513228 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:03.513228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:03.513396 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:03.513446 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:03.513447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.514188 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.514188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:03.514724 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:03.514726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.515107 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.515107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:03.515147 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:03.520009 (+ 4862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:03.520016 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:03.522384 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260925 05:11:03.522569 12285 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:03.512929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:53983 (local address 127.3.6.190:44825)
0925 05:11:03.513078 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:03.513080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:03.513092 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:11:03.513119 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:03.513120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:03.513138 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:03.513211 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:03.513554 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.514096 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.514813 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.515030 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.515099 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:03.522493 (+ 7394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:53983: BlockingRecv error: recv got EOF from 127.3.6.129:53983 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
Sep 25 05:11:03 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:03 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:03.643052 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:03.637803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:35327)
0925 05:11:03.638005 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:03.638020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:03.638114 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:03.638276 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:03.638276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:03.638498 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:03.638560 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:03.638563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.639569 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.639571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:03.640250 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:03.640253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.640658 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.640659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:03.640717 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:03.641148 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:03.641152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:03.642974 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260925 05:11:03.643191 12285 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:03.637906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:35327 (local address 127.3.6.190:44825)
0925 05:11:03.637981 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:03.637983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:03.638086 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:11:03.638186 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:03.638187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:03.638201 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:03.638271 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:03.638648 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.639476 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.640342 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.640561 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.640656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:03.643105 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:35327: BlockingRecv error: recv got EOF from 127.3.6.130:35327 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 05:11:03 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:03 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.131@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:03.782591 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:03.777765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.131:52369)
0925 05:11:03.777964 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:03.777977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:03.778084 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:03.778241 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:03.778242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:03.778432 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:03.778480 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:03.778482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.779167 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.779169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:03.779707 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:03.779709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.780120 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.780121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:03.780178 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:03.780692 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:03.780698 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:03.782490 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
W20260925 05:11:03.782660 12285 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:03.777868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:52369 (local address 127.3.6.190:44825)
0925 05:11:03.777963 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:03.777966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:03.778065 (+ 99us) server_negotiation.cc:407] Connection header received
0925 05:11:03.778144 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:03.778145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:03.778166 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:03.778234 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:03.778568 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.779071 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.779844 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:03.780032 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:03.780119 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:03.782577 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.131:52369: BlockingRecv error: recv got EOF from 127.3.6.131:52369 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 05:11:04 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:04 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.129@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:04.528712 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.523304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.129:34901)
0925 05:11:04.523492 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:04.523506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:04.523573 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:04.523894 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:04.523895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:04.524156 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:04.524219 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.524222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.525247 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.525249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.525790 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.525792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.526141 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.526142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.526186 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.526643 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:04.526648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:04.528648 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
W20260925 05:11:04.528846 12289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:04.523398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:34901 (local address 127.3.6.190:44825)
0925 05:11:04.523669 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:04.523672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:04.523682 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:11:04.523713 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:04.523713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:04.523728 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:04.523845 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:04.524337 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.525118 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.525871 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.526068 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.526134 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.528732 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.129:34901: BlockingRecv error: recv got EOF from 127.3.6.129:34901 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
Sep 25 05:11:04 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:11:04 dist-test-slave-1lvv krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.6.130@KRBTEST.COM for oryx/127.3.6.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:11:04.648309 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.643570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44825 (local address 127.3.6.130:45393)
0925 05:11:04.643706 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:04.643719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:04.643816 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:04.644006 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:04.644007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:04.644201 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:04.644257 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.644259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.644909 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.644910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.645442 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:04.645443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.645787 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.645788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:04.645828 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.646369 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:04.646373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:04.648236 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.6.190:44825: Server oryx/127.3.6.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":104,"threads_started":1}
W20260925 05:11:04.648473 12289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:04.643662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:45393 (local address 127.3.6.190:44825)
0925 05:11:04.643722 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:04.643726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:04.643803 (+ 77us) server_negotiation.cc:407] Connection header received
0925 05:11:04.643868 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:04.643869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:04.643879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:04.643937 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:04.644337 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.644828 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.645521 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.645717 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.645774 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.648387 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.6.130:45393: BlockingRecv error: recv got EOF from 127.3.6.130:45393 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:11:04.695595 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 11814
I20260925 05:11:04.700914 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 11966
I20260925 05:11:04.705700 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 12098
I20260925 05:11:04.710458 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 11744
2026-09-25T05:11:04Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17589 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T05:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:04Z Disabled control of system clock
I20260925 05:11:04.740635 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:39999
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:11:04.822844 12294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:04.823084 12294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:04.823107 12294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:04.824316 12294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:04.824360 12294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:04.824373 12294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:04.824386 12294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:04.824399 12294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:04.825718 12294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:39999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:39999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:04.825929 12294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:04.826192 12294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:04.831390 12300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:04.831460 12294 server_base.cc:1034] running on GCE node
W20260925 05:11:04.831390 12302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:04.831408 12299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:04.832052 12294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:04.832350 12294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:04.833662 12294 hybrid_clock.cc:648] HybridClock initialized: now 1790313064833644 us; error 54 us; skew 500 ppm
I20260925 05:11:04.835227 12294 webserver.cc:460] Webserver started at http://127.3.6.190:34345/ using document root <none> and password file <none>
I20260925 05:11:04.835543 12294 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:04.835625 12294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:04.835848 12294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:04.837040 12294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "6995fa94d1124df394003c0e9654adb8"
format_stamp: "Formatted at 2026-09-25 05:11:04 on dist-test-slave-1lvv"
server_key: "df9754ac9ef3054d24a84c5ae7ca8926"
server_key_iv: "1a7b88c56ea0a3936a75719f0149262e"
server_key_version: "encryptionkey@0"
I20260925 05:11:04.837445 12294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "6995fa94d1124df394003c0e9654adb8"
format_stamp: "Formatted at 2026-09-25 05:11:04 on dist-test-slave-1lvv"
server_key: "df9754ac9ef3054d24a84c5ae7ca8926"
server_key_iv: "1a7b88c56ea0a3936a75719f0149262e"
server_key_version: "encryptionkey@0"
I20260925 05:11:04.838984 12294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:11:04.839883 12308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:04.840075 12294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:04.840184 12294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "6995fa94d1124df394003c0e9654adb8"
format_stamp: "Formatted at 2026-09-25 05:11:04 on dist-test-slave-1lvv"
server_key: "df9754ac9ef3054d24a84c5ae7ca8926"
server_key_iv: "1a7b88c56ea0a3936a75719f0149262e"
server_key_version: "encryptionkey@0"
I20260925 05:11:04.840281 12294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:04.858431 12294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:04.858816 12294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:04.858985 12294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:04.863113 12294 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:39999
I20260925 05:11:04.863173 12360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:39999 every 8 connection(s)
I20260925 05:11:04.863564 12294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:04.864365 12361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:04.865329 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 12294
I20260925 05:11:04.865406 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:11:04.865633 3098 external_mini_cluster.cc:1428] Setting key f5bd7e86b4d92f670e826670cde0a30c
I20260925 05:11:04.867460 12361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8: Bootstrap starting.
I20260925 05:11:04.868186 12361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:04.868506 12361 log.cc:824] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:04.869218 12361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8: No bootstrap required, opened a new log
I20260925 05:11:04.871158 12361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6995fa94d1124df394003c0e9654adb8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39999 } }
I20260925 05:11:04.871330 12361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:04.871393 12361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6995fa94d1124df394003c0e9654adb8, State: Initialized, Role: FOLLOWER
I20260925 05:11:04.871564 12361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6995fa94d1124df394003c0e9654adb8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39999 } }
I20260925 05:11:04.871675 12361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:04.871734 12361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:04.871824 12361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:04.872403 12361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6995fa94d1124df394003c0e9654adb8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39999 } }
I20260925 05:11:04.872494 12361 leader_election.cc:302] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6995fa94d1124df394003c0e9654adb8; no voters:
I20260925 05:11:04.872700 12361 leader_election.cc:288] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:04.872743 12366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:04.873041 12366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [term 1 LEADER]: Becoming Leader. State: Replica: 6995fa94d1124df394003c0e9654adb8, State: Running, Role: LEADER
I20260925 05:11:04.873106 12361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:04.873191 12366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6995fa94d1124df394003c0e9654adb8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39999 } }
I20260925 05:11:04.873510 12366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6995fa94d1124df394003c0e9654adb8. Latest consensus state: current_term: 1 leader_uuid: "6995fa94d1124df394003c0e9654adb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6995fa94d1124df394003c0e9654adb8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39999 } } }
I20260925 05:11:04.873541 12367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6995fa94d1124df394003c0e9654adb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6995fa94d1124df394003c0e9654adb8" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39999 } } }
I20260925 05:11:04.873072 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:04.866280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.3.6.190:39999)
0925 05:11:04.866485 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:04.866494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:04.868043 (+ 1549us) server_negotiation.cc:407] Connection header received
0925 05:11:04.868580 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:04.868588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:04.868688 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:04.868835 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:04.869076 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.870032 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.870898 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:04.871312 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:04.871458 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:04.872285 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:04.872289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:04.872290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:04.872495 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:04.872538 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:04.872553 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:04.872628 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:04.872753 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 05:11:04.872844 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260925 05:11:04.873628 12366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:04.873651 12367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:04.874118 12372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:04.874874 12372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:04.884547 12372 catalog_manager.cc:1347] Generated new cluster ID: af798ef075494d29a46f71f24b696db1
I20260925 05:11:04.884640 12372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:04.890607 12372 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:04.891145 12372 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:04.900270 12372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6995fa94d1124df394003c0e9654adb8: Generated new TSK 0
I20260925 05:11:04.900508 12372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:11:04.902251 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:39999
--builtin_ntp_servers=127.3.6.148:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 05:11:04.989238 12385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:04.989527 12385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:04.989569 12385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:04.990875 12385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:04.990962 12385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:04.991006 12385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:11:04.992597 12385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:04.992906 12385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:04.993263 12385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:04.998469 12391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:04.998456 12393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:04.999153 12390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:04.999111 12385 server_base.cc:1034] running on GCE node
I20260925 05:11:04.999449 12385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:04.999799 12385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:05.000973 12385 hybrid_clock.cc:648] HybridClock initialized: now 1790313065000952 us; error 47 us; skew 500 ppm
I20260925 05:11:05.002501 12385 webserver.cc:460] Webserver started at http://127.3.6.129:35831/ using document root <none> and password file <none>
I20260925 05:11:05.002761 12385 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:05.002835 12385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:05.002983 12385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:05.004210 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "9b2cd485daea41c59a345cd9729785ee"
format_stamp: "Formatted at 2026-09-25 05:11:04 on dist-test-slave-1lvv"
server_key: "f78c82b065e759c50c414f2ba0eda4ea"
server_key_iv: "f25d0f9b94bfbc87b76a420982fb11e2"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.004606 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "9b2cd485daea41c59a345cd9729785ee"
format_stamp: "Formatted at 2026-09-25 05:11:04 on dist-test-slave-1lvv"
server_key: "f78c82b065e759c50c414f2ba0eda4ea"
server_key_iv: "f25d0f9b94bfbc87b76a420982fb11e2"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.006057 12385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:11:05.006898 12399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.007097 12385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:05.007206 12385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "9b2cd485daea41c59a345cd9729785ee"
format_stamp: "Formatted at 2026-09-25 05:11:04 on dist-test-slave-1lvv"
server_key: "f78c82b065e759c50c414f2ba0eda4ea"
server_key_iv: "f25d0f9b94bfbc87b76a420982fb11e2"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.007315 12385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:05.020656 12385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:05.021028 12385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:05.021205 12385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:05.021493 12385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:05.021962 12385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:05.022022 12385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.022102 12385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:05.022149 12385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.030260 12385 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:44175
I20260925 05:11:05.030293 12512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:44175 every 8 connection(s)
I20260925 05:11:05.030763 12385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:05.036799 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.031814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:42173 (local address 127.3.6.190:39999)
0925 05:11:05.031864 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.031867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.032176 (+ 309us) server_negotiation.cc:407] Connection header received
0925 05:11:05.032804 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.032805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.032817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.032874 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.033582 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.034103 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.035169 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.035393 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.035478 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.036489 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.036490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.036491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:05.036497 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.036513 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:05.036517 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.036571 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.036675 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.036712 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:11:05.037066 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.031824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:39999 (local address 127.3.6.129:42173)
0925 05:11:05.032023 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.032066 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.032690 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.032955 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.032959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.032970 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:05.033479 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.033484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.034228 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.034229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.035064 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.035066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.035488 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.035489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.035644 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.036316 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:05.036317 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:05.036350 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:05.036378 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:05.036580 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:05.036585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:05.036588 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.036678 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.036793 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260925 05:11:05.037817 12513 heartbeater.cc:347] Connected to a master server at 127.3.6.190:39999
I20260925 05:11:05.037943 12513 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:05.038144 12513 heartbeater.cc:511] Master 127.3.6.190:39999 requested a full tablet report, sending...
I20260925 05:11:05.038547 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 12385
I20260925 05:11:05.038646 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:05.038744 12325 ts_manager.cc:194] Registered new tserver with Master: 9b2cd485daea41c59a345cd9729785ee (127.3.6.129:44175)
I20260925 05:11:05.038862 3098 external_mini_cluster.cc:1428] Setting key dda6a89a4fcd73ef266b65018ac78ec0
I20260925 05:11:05.039492 12325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.129:42173
I20260925 05:11:05.040608 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:39999
--builtin_ntp_servers=127.3.6.148:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 05:11:05.128347 12516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:05.128613 12516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:05.128651 12516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:05.129870 12516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:05.129941 12516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:05.129985 12516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:11:05.131385 12516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:05.131681 12516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:05.132033 12516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:05.138206 12521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.138206 12522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:05.138554 12516 server_base.cc:1034] running on GCE node
W20260925 05:11:05.138306 12524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:05.138939 12516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:05.139249 12516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:05.140416 12516 hybrid_clock.cc:648] HybridClock initialized: now 1790313065140393 us; error 45 us; skew 500 ppm
I20260925 05:11:05.141745 12516 webserver.cc:460] Webserver started at http://127.3.6.130:38535/ using document root <none> and password file <none>
I20260925 05:11:05.142014 12516 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:05.142092 12516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:05.142246 12516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:05.143663 12516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "ba7755db19f149bd8b0021a31d2fe1ad"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "97ab5582c843afa1b355a96705703ef2"
server_key_iv: "75c29079f94b20d119ed7758b6202b2f"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.144129 12516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "ba7755db19f149bd8b0021a31d2fe1ad"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "97ab5582c843afa1b355a96705703ef2"
server_key_iv: "75c29079f94b20d119ed7758b6202b2f"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.145661 12516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:11:05.146476 12530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.146669 12516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:05.146771 12516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "ba7755db19f149bd8b0021a31d2fe1ad"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "97ab5582c843afa1b355a96705703ef2"
server_key_iv: "75c29079f94b20d119ed7758b6202b2f"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.146848 12516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:05.171492 12516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:05.171833 12516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:05.171952 12516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:05.172169 12516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:05.172559 12516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:05.172595 12516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.172653 12516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:05.172680 12516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.178851 12516 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:42863
I20260925 05:11:05.178893 12643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:42863 every 8 connection(s)
I20260925 05:11:05.179263 12516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:05.184916 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.180181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:58567 (local address 127.3.6.190:39999)
0925 05:11:05.180258 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.180260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.180547 (+ 287us) server_negotiation.cc:407] Connection header received
0925 05:11:05.181084 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.181085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.181096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.181176 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.181871 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.182364 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.183379 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.183604 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.183664 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.184673 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.184674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.184674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:05.184680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.184691 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:05.184695 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.184739 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.184821 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.184851 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:11:05.185178 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.180142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:39999 (local address 127.3.6.130:58567)
0925 05:11:05.180374 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.180418 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.180983 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.181262 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.181265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.181274 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:05.181787 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.181791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.182456 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.182457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.183288 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.183291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.183694 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.183695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.183905 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.184535 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:05.184535 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:05.184568 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:05.184592 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:05.184752 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:05.184756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:05.184759 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.184813 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.184905 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":117,"threads_started":1}
I20260925 05:11:05.185752 12644 heartbeater.cc:347] Connected to a master server at 127.3.6.190:39999
I20260925 05:11:05.185864 12644 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:05.186065 12644 heartbeater.cc:511] Master 127.3.6.190:39999 requested a full tablet report, sending...
I20260925 05:11:05.186481 12325 ts_manager.cc:194] Registered new tserver with Master: ba7755db19f149bd8b0021a31d2fe1ad (127.3.6.130:42863)
I20260925 05:11:05.186861 12325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.130:58567
I20260925 05:11:05.187088 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 12516
I20260925 05:11:05.187207 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:05.187412 3098 external_mini_cluster.cc:1428] Setting key bd817fa8e269858b997f834d2f5a14d8
I20260925 05:11:05.188283 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:39999
--builtin_ntp_servers=127.3.6.148:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 05:11:05.273082 12647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:05.273381 12647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:05.273417 12647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:05.274796 12647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:05.274842 12647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:05.274869 12647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:11:05.276330 12647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:05.276544 12647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:05.276825 12647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:05.281987 12652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.282003 12653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.281987 12655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:05.282281 12647 server_base.cc:1034] running on GCE node
I20260925 05:11:05.282531 12647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:05.282877 12647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:05.284050 12647 hybrid_clock.cc:648] HybridClock initialized: now 1790313065284043 us; error 41 us; skew 500 ppm
I20260925 05:11:05.285588 12647 webserver.cc:460] Webserver started at http://127.3.6.131:37815/ using document root <none> and password file <none>
I20260925 05:11:05.285867 12647 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:05.285943 12647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:05.286101 12647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:05.287240 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "ab545fe93c54438882d95581f9b407f5"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "9affe85225a2815728167e991a5f1d0d"
server_key_iv: "e057d979130e0577e0f12849f83d6591"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.287627 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "ab545fe93c54438882d95581f9b407f5"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "9affe85225a2815728167e991a5f1d0d"
server_key_iv: "e057d979130e0577e0f12849f83d6591"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.289075 12647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:11:05.289840 12661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.290012 12647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:11:05.290123 12647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "ab545fe93c54438882d95581f9b407f5"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "9affe85225a2815728167e991a5f1d0d"
server_key_iv: "e057d979130e0577e0f12849f83d6591"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.290196 12647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:05.302922 12647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:05.303292 12647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:05.303462 12647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:05.303711 12647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:05.304119 12647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:05.304185 12647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.304224 12647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:05.304270 12647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.310261 12647 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:44803
I20260925 05:11:05.310351 12774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:44803 every 8 connection(s)
I20260925 05:11:05.310806 12647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:05.313930 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 12647
I20260925 05:11:05.314028 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:05.314248 3098 external_mini_cluster.cc:1428] Setting key b0d5c2780f88ab7d023c54b330753727
I20260925 05:11:05.317054 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.311554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:58617 (local address 127.3.6.190:39999)
0925 05:11:05.311625 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.311627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.311946 (+ 319us) server_negotiation.cc:407] Connection header received
0925 05:11:05.312571 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.312572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.312585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.312649 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.313408 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.314122 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.315576 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.315830 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.315904 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.316796 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.316797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.316797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:05.316802 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.316818 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:05.316822 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.316868 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.316959 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.316987 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:11:05.317327 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.311561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:39999 (local address 127.3.6.131:58617)
0925 05:11:05.311806 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.311855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.312427 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.312731 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.312735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.312752 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:05.313286 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.313290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.314244 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.314245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.315483 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:05.315485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:05.315933 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:05.315934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:05.316075 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:05.316638 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:05.316638 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:05.316670 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:05.316697 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:05.316881 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:05.316885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:05.316888 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.316946 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.317047 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260925 05:11:05.317983 12775 heartbeater.cc:347] Connected to a master server at 127.3.6.190:39999
I20260925 05:11:05.318104 12775 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:05.318325 12775 heartbeater.cc:511] Master 127.3.6.190:39999 requested a full tablet report, sending...
I20260925 05:11:05.318717 12325 ts_manager.cc:194] Registered new tserver with Master: ab545fe93c54438882d95581f9b407f5 (127.3.6.131:44803)
I20260925 05:11:05.319068 12325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.131:58617
I20260925 05:11:05.325977 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:11:05.327971 12364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:05.327395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.3.6.190:39999)
0925 05:11:05.327492 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.327494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.327601 (+ 107us) server_negotiation.cc:407] Connection header received
0925 05:11:05.327685 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.327686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.327696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.327743 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.327893 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39860: BlockingRecv error: recv got EOF from 127.0.0.1:39860 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:11:05.328001 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 12385
I20260925 05:11:05.333189 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 12516
I20260925 05:11:05.337838 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 12647
I20260925 05:11:05.342545 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 12294
2026-09-25T05:11:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (631 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T05:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:05Z Disabled control of system clock
I20260925 05:11:05.371816 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44817
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:11:05.455174 12786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:05.455413 12786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:05.455435 12786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:05.456674 12786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:05.456720 12786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:05.456734 12786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:05.456746 12786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:05.456758 12786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:05.458078 12786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:05.458287 12786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:05.458549 12786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:11:05.463598 12786 server_base.cc:1034] running on GCE node
W20260925 05:11:05.463568 12791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.463611 12792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.463568 12794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:05.464053 12786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:05.464378 12786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:05.465533 12786 hybrid_clock.cc:648] HybridClock initialized: now 1790313065465510 us; error 37 us; skew 500 ppm
I20260925 05:11:05.466943 12786 webserver.cc:460] Webserver started at http://127.3.6.190:39317/ using document root <none> and password file <none>
I20260925 05:11:05.467238 12786 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:05.467293 12786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:05.467433 12786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:05.468621 12786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "707c3bd7d80d4c66a31c4cf9a97fb340"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "15b282ae831a7a559a6d8e7a6a447afd"
server_key_iv: "e4bf8261bae567570a3d354a268bd5b3"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.469049 12786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "707c3bd7d80d4c66a31c4cf9a97fb340"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "15b282ae831a7a559a6d8e7a6a447afd"
server_key_iv: "e4bf8261bae567570a3d354a268bd5b3"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.470516 12786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:11:05.471494 12800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.471758 12786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:05.471908 12786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "707c3bd7d80d4c66a31c4cf9a97fb340"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "15b282ae831a7a559a6d8e7a6a447afd"
server_key_iv: "e4bf8261bae567570a3d354a268bd5b3"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.472004 12786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:05.475405 12786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:05.475701 12786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:05.475883 12786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:05.479820 12786 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44817
I20260925 05:11:05.479869 12852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44817 every 8 connection(s)
I20260925 05:11:05.480309 12786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:05.480993 12853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:05.483923 12853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340: Bootstrap starting.
I20260925 05:11:05.484526 12853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:05.484783 12853 log.cc:824] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:05.485479 12853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340: No bootstrap required, opened a new log
I20260925 05:11:05.486677 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 12786
I20260925 05:11:05.486749 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:11:05.486955 3098 external_mini_cluster.cc:1428] Setting key 3f98a884a930507fb047a450406e50d7
I20260925 05:11:05.487342 12853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44817 } }
I20260925 05:11:05.487572 12853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:05.487674 12853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707c3bd7d80d4c66a31c4cf9a97fb340, State: Initialized, Role: FOLLOWER
I20260925 05:11:05.487869 12853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44817 } }
I20260925 05:11:05.487973 12853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:05.488004 12853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:05.488057 12853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:05.488835 12853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44817 } }
I20260925 05:11:05.488966 12853 leader_election.cc:302] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707c3bd7d80d4c66a31c4cf9a97fb340; no voters:
I20260925 05:11:05.489135 12853 leader_election.cc:288] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:05.489190 12858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:05.489427 12858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [term 1 LEADER]: Becoming Leader. State: Replica: 707c3bd7d80d4c66a31c4cf9a97fb340, State: Running, Role: LEADER
I20260925 05:11:05.489532 12853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:05.489558 12858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44817 } }
I20260925 05:11:05.489768 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.487570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.3.6.190:44817)
0925 05:11:05.487879 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.487889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.487922 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:11:05.488060 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.488068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.488120 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.488272 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.488946 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.488950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.488950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:05.488955 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.489005 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:05.489030 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.489109 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.489209 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.489273 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260925 05:11:05.490182 12860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707c3bd7d80d4c66a31c4cf9a97fb340. Latest consensus state: current_term: 1 leader_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44817 } } }
I20260925 05:11:05.490291 12860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:05.490170 12859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c3bd7d80d4c66a31c4cf9a97fb340" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44817 } } }
I20260925 05:11:05.490518 12867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:05.490569 12859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:05.491065 12867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:05.492766 12867 catalog_manager.cc:1347] Generated new cluster ID: 23dfade475974c01854067b2b9b00d68
I20260925 05:11:05.492825 12867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:05.510185 12867 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:05.510397 12867 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:05.515385 12867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707c3bd7d80d4c66a31c4cf9a97fb340: Generated new TSK 0
I20260925 05:11:05.515645 12867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:11:05.523561 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44817
--builtin_ntp_servers=127.3.6.148:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:11:05.612339 12877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:05.612601 12877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:05.612641 12877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:05.613917 12877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:05.614001 12877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:05.614046 12877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:11:05.615513 12877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:05.615885 12877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:05.616246 12877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:05.622419 12885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.622442 12883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.622460 12882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:05.623215 12877 server_base.cc:1034] running on GCE node
I20260925 05:11:05.623488 12877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:05.623812 12877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:05.625003 12877 hybrid_clock.cc:648] HybridClock initialized: now 1790313065624987 us; error 38 us; skew 500 ppm
I20260925 05:11:05.626528 12877 webserver.cc:460] Webserver started at http://127.3.6.129:41085/ using document root <none> and password file <none>
I20260925 05:11:05.626845 12877 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:05.626928 12877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:05.627095 12877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:05.628449 12877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "3820142df752483aa993776c89305918"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "f9e7ff43b93e9860c215050818660958"
server_key_iv: "bf6ce3b08d8a158b91df51cc6d1bd192"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.628861 12877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "3820142df752483aa993776c89305918"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "f9e7ff43b93e9860c215050818660958"
server_key_iv: "bf6ce3b08d8a158b91df51cc6d1bd192"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.630324 12877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:11:05.631456 12891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.631649 12877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:05.631799 12877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "3820142df752483aa993776c89305918"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "f9e7ff43b93e9860c215050818660958"
server_key_iv: "bf6ce3b08d8a158b91df51cc6d1bd192"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.631887 12877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:05.637301 12877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:05.637698 12877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:05.637859 12877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:05.638090 12877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:05.638454 12877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:05.638510 12877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.638571 12877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:05.638603 12877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.646528 12877 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:42687
I20260925 05:11:05.646576 13004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:42687 every 8 connection(s)
I20260925 05:11:05.647023 12877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:05.650794 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.648136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:34339 (local address 127.3.6.190:44817)
0925 05:11:05.648189 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.648191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.648435 (+ 244us) server_negotiation.cc:407] Connection header received
0925 05:11:05.649100 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.649102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.649112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.649202 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.650474 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.650476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.650476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:05.650481 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.650500 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:05.650504 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.650563 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.650698 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.650719 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:11:05.651003 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.647876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44817 (local address 127.3.6.129:34339)
0925 05:11:05.648286 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.648334 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.648971 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.649302 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.649330 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.649343 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:05.650301 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:05.650301 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:05.650333 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:05.650365 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:05.650598 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:05.650604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:05.650607 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.650711 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.650779 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":95,"threads_started":1}
I20260925 05:11:05.651216 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 12877
I20260925 05:11:05.651319 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:05.651554 3098 external_mini_cluster.cc:1428] Setting key d3cdd5699314b24ae83f2f22324c2372
I20260925 05:11:05.651666 13005 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44817
I20260925 05:11:05.651794 13005 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:05.651959 13005 heartbeater.cc:511] Master 127.3.6.190:44817 requested a full tablet report, sending...
I20260925 05:11:05.652501 12817 ts_manager.cc:194] Registered new tserver with Master: 3820142df752483aa993776c89305918 (127.3.6.129:42687)
I20260925 05:11:05.652611 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44817
--builtin_ntp_servers=127.3.6.148:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:11:05.741360 13008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:05.741611 13008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:05.741636 13008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:05.742863 13008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:05.742909 13008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:05.742937 13008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:11:05.744434 13008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:05.744670 13008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:05.745028 13008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:05.750615 13016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.750618 13013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.750618 13014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:05.751015 13008 server_base.cc:1034] running on GCE node
I20260925 05:11:05.751298 13008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:05.751571 13008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:05.752718 13008 hybrid_clock.cc:648] HybridClock initialized: now 1790313065752699 us; error 46 us; skew 500 ppm
I20260925 05:11:05.754168 13008 webserver.cc:460] Webserver started at http://127.3.6.130:38955/ using document root <none> and password file <none>
I20260925 05:11:05.754443 13008 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:05.754495 13008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:05.754664 13008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:05.755874 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "37a31c19d38d4ec0a823c7b1036ec286"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "356f08ebe8471b9abce430e4653182ff"
server_key_iv: "69bf46ff1cf416643103b5419bf73a74"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.756270 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "37a31c19d38d4ec0a823c7b1036ec286"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "356f08ebe8471b9abce430e4653182ff"
server_key_iv: "69bf46ff1cf416643103b5419bf73a74"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.757783 13008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:11:05.758579 13022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.758785 13008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:11:05.758882 13008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "37a31c19d38d4ec0a823c7b1036ec286"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "356f08ebe8471b9abce430e4653182ff"
server_key_iv: "69bf46ff1cf416643103b5419bf73a74"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.758965 13008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:05.762609 13008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:05.762904 13008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:05.763056 13008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:05.763267 13008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:05.763600 13008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:05.763661 13008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.763702 13008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:05.763731 13008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.769680 13008 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:40619
I20260925 05:11:05.769721 13135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:40619 every 8 connection(s)
I20260925 05:11:05.770114 13008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:05.773301 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.770934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:60847 (local address 127.3.6.190:44817)
0925 05:11:05.770982 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.770984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.771232 (+ 248us) server_negotiation.cc:407] Connection header received
0925 05:11:05.771717 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.771718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.771724 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.771862 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.773073 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.773075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.773075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:05.773079 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.773095 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:05.773099 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.773154 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.773235 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.773248 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:11:05.773527 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.770923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44817 (local address 127.3.6.130:60847)
0925 05:11:05.771100 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.771134 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.771628 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.771923 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.771926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.771935 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:05.772918 (+ 983us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:05.772918 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:05.772948 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:05.772977 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:05.773165 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:05.773169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:05.773172 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.773232 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.773283 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260925 05:11:05.774039 13136 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44817
I20260925 05:11:05.774147 13136 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:05.774315 13136 heartbeater.cc:511] Master 127.3.6.190:44817 requested a full tablet report, sending...
I20260925 05:11:05.774690 12817 ts_manager.cc:194] Registered new tserver with Master: 37a31c19d38d4ec0a823c7b1036ec286 (127.3.6.130:40619)
I20260925 05:11:05.778102 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 13008
I20260925 05:11:05.778182 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:05.778391 3098 external_mini_cluster.cc:1428] Setting key 1f4522c1c26d31b096ce1ace4f1ba8d5
I20260925 05:11:05.779286 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44817
--builtin_ntp_servers=127.3.6.148:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:11:05.866797 13139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:05.867056 13139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:05.867082 13139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:05.868988 13139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:05.869067 13139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:05.869117 13139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:11:05.871102 13139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:05.871431 13139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:05.871873 13139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:05.877939 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.877939 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:05.877942 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:05.878189 13139 server_base.cc:1034] running on GCE node
I20260925 05:11:05.878489 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:05.878779 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:05.879948 13139 hybrid_clock.cc:648] HybridClock initialized: now 1790313065879935 us; error 33 us; skew 500 ppm
I20260925 05:11:05.881438 13139 webserver.cc:460] Webserver started at http://127.3.6.131:38143/ using document root <none> and password file <none>
I20260925 05:11:05.881731 13139 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:05.881806 13139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:05.881973 13139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:05.883131 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "dda0419bb293405bb6baf0607cc29234"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "c862e048cd894d8a015503964de667b3"
server_key_iv: "ba4a55267e90c27b6e56e865b5e14b04"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.883525 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "dda0419bb293405bb6baf0607cc29234"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "c862e048cd894d8a015503964de667b3"
server_key_iv: "ba4a55267e90c27b6e56e865b5e14b04"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.885170 13139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:11:05.886049 13153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.886228 13139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:05.886343 13139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "dda0419bb293405bb6baf0607cc29234"
format_stamp: "Formatted at 2026-09-25 05:11:05 on dist-test-slave-1lvv"
server_key: "c862e048cd894d8a015503964de667b3"
server_key_iv: "ba4a55267e90c27b6e56e865b5e14b04"
server_key_version: "encryptionkey@0"
I20260925 05:11:05.886425 13139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:05.890295 13139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:05.890601 13139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:05.890762 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:05.891000 13139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:05.891330 13139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:05.891384 13139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.891426 13139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:05.891453 13139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:05.897384 13139 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:37041
I20260925 05:11:05.897428 13266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:37041 every 8 connection(s)
I20260925 05:11:05.897825 13139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:05.901182 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.898573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:54817 (local address 127.3.6.190:44817)
0925 05:11:05.898650 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.898652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.898894 (+ 242us) server_negotiation.cc:407] Connection header received
0925 05:11:05.899423 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.899425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.899432 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.899562 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.900903 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:05.900904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:05.900904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:11:05.900909 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:05.900925 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:11:05.900930 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:05.901005 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:05.901088 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 05:11:05.901105 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:11:05.901419 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:05.898571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44817 (local address 127.3.6.131:54817)
0925 05:11:05.898767 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:05.898802 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:05.899307 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:05.899651 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:05.899655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:05.899663 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:05.900723 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:11:05.900724 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:05.900755 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:11:05.900786 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:05.901003 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:05.901010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:05.901013 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:11:05.901100 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:11:05.901167 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260925 05:11:05.901989 13267 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44817
I20260925 05:11:05.902071 13267 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:05.902245 13267 heartbeater.cc:511] Master 127.3.6.190:44817 requested a full tablet report, sending...
I20260925 05:11:05.902622 12817 ts_manager.cc:194] Registered new tserver with Master: dda0419bb293405bb6baf0607cc29234 (127.3.6.131:37041)
I20260925 05:11:05.905031 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 13139
I20260925 05:11:05.905159 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:05.905386 3098 external_mini_cluster.cc:1428] Setting key e248ca62e7a367a02b7f29bc67cc4d99
I20260925 05:11:05.906478 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:11:05.908460 12857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:11:05.907897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.3.6.190:44817)
0925 05:11:05.907972 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:05.907974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:05.908124 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:11:05.908194 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:05.908195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:05.908201 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:05.908257 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:05.908383 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51644: BlockingRecv error: recv got EOF from 127.0.0.1:51644 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:11:05.908500 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 12877
I20260925 05:11:05.913374 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 13008
I20260925 05:11:05.918987 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 13139
I20260925 05:11:05.923863 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 12786
2026-09-25T05:11:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (580 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:11:05 dist-test-slave-1lvv krb5kdc[13277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:11:05 dist-test-slave-1lvv krb5kdc[13277](info): set up 2 sockets
Sep 25 05:11:05 dist-test-slave-1lvv krb5kdc[13277](info): commencing operation
krb5kdc: starting...
W20260925 05:11:06.977210 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:11:07.108906 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40031
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:07.192150 13290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:07.192431 13290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:07.192469 13290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:07.193652 13290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:07.193730 13290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:07.193755 13290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:07.193775 13290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:07.193809 13290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:07.195331 13290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:07.195634 13290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:07.195905 13290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:07.201170 13295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:07.201222 13296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:07.201195 13298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:07.201586 13290 server_base.cc:1034] running on GCE node
I20260925 05:11:07.201846 13290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:07.202154 13290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:07.203333 13290 hybrid_clock.cc:648] HybridClock initialized: now 1790313067203320 us; error 45 us; skew 500 ppm
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:07.205953 13290 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:11:07.206427 13290 webserver.cc:460] Webserver started at http://127.3.6.190:35291/ using document root <none> and password file <none>
I20260925 05:11:07.206667 13290 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:07.206738 13290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:07.206895 13290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:07.208084 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "f42ca3893e2a4b59a6d908d6f089ae2f"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "ae94abc3600d6376262b16347be34e59"
server_key_iv: "04c67945c61c3f2c35be7495f6713b4d"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.208483 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "f42ca3893e2a4b59a6d908d6f089ae2f"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "ae94abc3600d6376262b16347be34e59"
server_key_iv: "04c67945c61c3f2c35be7495f6713b4d"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.209838 13290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:07.210572 13305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.210757 13290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:11:07.210865 13290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "f42ca3893e2a4b59a6d908d6f089ae2f"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "ae94abc3600d6376262b16347be34e59"
server_key_iv: "04c67945c61c3f2c35be7495f6713b4d"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.210938 13290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:07.226123 13290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:07.228746 13290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:07.229074 13290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:07.233654 13290 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:40031
I20260925 05:11:07.233688 13357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:40031 every 8 connection(s)
I20260925 05:11:07.234120 13290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:07.234519 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 13290
I20260925 05:11:07.234660 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:11:07.234879 3098 external_mini_cluster.cc:1428] Setting key 84be81e94a27495c0c013c1e51c96473
I20260925 05:11:07.235354 13358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:07.238526 13358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f: Bootstrap starting.
I20260925 05:11:07.239185 13358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:07.239517 13358 log.cc:824] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:07.240378 13358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f: No bootstrap required, opened a new log
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:11:07.243244 13358 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40031 } }
I20260925 05:11:07.243449 13358 raft_consensus.cc:374] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:07.243518 13358 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42ca3893e2a4b59a6d908d6f089ae2f, State: Initialized, Role: FOLLOWER
I20260925 05:11:07.243659 13358 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40031 } }
I20260925 05:11:07.243798 13358 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:07.243848 13358 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:07.243916 13358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:07.244596 13358 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40031 } }
I20260925 05:11:07.244738 13358 leader_election.cc:302] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42ca3893e2a4b59a6d908d6f089ae2f; no voters:
I20260925 05:11:07.244927 13358 leader_election.cc:288] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:07.244983 13363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:07.245160 13363 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [term 1 LEADER]: Becoming Leader. State: Replica: f42ca3893e2a4b59a6d908d6f089ae2f, State: Running, Role: LEADER
I20260925 05:11:07.245251 13358 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:07.245265 13363 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40031 } }
I20260925 05:11:07.245900 13365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40031 } } }
I20260925 05:11:07.245919 13364 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42ca3893e2a4b59a6d908d6f089ae2f. Latest consensus state: current_term: 1 leader_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42ca3893e2a4b59a6d908d6f089ae2f" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40031 } } }
I20260925 05:11:07.246119 13365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:07.246136 13364 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:07.246789 13373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:07.247154 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.235538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.3.6.190:40031)
0925 05:11:07.235815 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.235829 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.235859 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:11:07.236569 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.236576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.236626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.236783 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:07.237416 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.238299 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.239136 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.239604 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.239719 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.242964 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:07.242968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:07.242968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:07.242980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:07.244907 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.245363 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.245365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.245366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.245466 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.245698 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.245699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.245699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.246015 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:07.246173 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.246762 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.246923 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260925 05:11:07.247437 13373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:07.248975 13373 catalog_manager.cc:1347] Generated new cluster ID: d5367ec043f9462eb2076e3416f81675
I20260925 05:11:07.249066 13373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:07.258780 13373 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:07.259388 13373 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:07.268733 13373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f: Generated new TSK 0
I20260925 05:11:07.268992 13373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:11:07.302728 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40031
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:07.389261 13384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:07.389495 13384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:07.389520 13384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:07.390743 13384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:07.390795 13384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:07.390825 13384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:11:07.392354 13384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:07.392609 13384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:07.392882 13384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:07.398320 13389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:07.398409 13384 server_base.cc:1034] running on GCE node
W20260925 05:11:07.398502 13392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:07.398278 13390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:07.398833 13384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:07.399140 13384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:07.400300 13384 hybrid_clock.cc:648] HybridClock initialized: now 1790313067400280 us; error 49 us; skew 500 ppm
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:07.403285 13384 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:11:07.403941 13384 webserver.cc:460] Webserver started at http://127.3.6.129:41123/ using document root <none> and password file <none>
I20260925 05:11:07.404202 13384 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:07.404274 13384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:07.404431 13384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:07.405596 13384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "2bedfcfbb465476b9a62a531d415978b"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "8f5adabeafffac151b4e0c8d828f1fad"
server_key_iv: "94692b1d6114311c0e369297dda6cc35"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.406013 13384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "2bedfcfbb465476b9a62a531d415978b"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "8f5adabeafffac151b4e0c8d828f1fad"
server_key_iv: "94692b1d6114311c0e369297dda6cc35"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.407517 13384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:11:07.408497 13399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.408702 13384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:11:07.408811 13384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "2bedfcfbb465476b9a62a531d415978b"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "8f5adabeafffac151b4e0c8d828f1fad"
server_key_iv: "94692b1d6114311c0e369297dda6cc35"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.408897 13384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:07.418920 13384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:07.421597 13384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:07.421785 13384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:07.422043 13384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:07.422374 13384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:07.422428 13384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.422473 13384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:07.422500 13384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.428751 13384 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:39117
I20260925 05:11:07.428776 13512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:39117 every 8 connection(s)
I20260925 05:11:07.429168 13384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:11:07.438992 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.430036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:38821 (local address 127.3.6.190:40031)
0925 05:11:07.430111 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.430113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.430320 (+ 207us) server_negotiation.cc:407] Connection header received
0925 05:11:07.430808 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.430809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.430818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.430888 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:07.432036 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.432587 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.433651 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.433867 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.433943 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.436318 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:07.436320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:07.436320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:07.436327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:07.437564 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.437958 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.437958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.437959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.438013 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.438235 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.438235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.438235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.438344 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:07.438399 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.438771 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.438846 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:11:07.439114 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.429988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40031 (local address 127.3.6.129:38821)
0925 05:11:07.430170 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.430216 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.430716 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.430974 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.430978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.431608 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:07.431917 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.431925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.432680 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.432681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.433539 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.433541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.433957 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.433958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.434073 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.434673 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:07.434682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:07.436218 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:07.437649 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:07.437656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:07.437659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:07.437889 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:07.438072 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:07.438073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:07.438073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:07.438179 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:07.438495 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:07.438505 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:07.438646 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.438745 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.438884 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260925 05:11:07.439103 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 13384
I20260925 05:11:07.439199 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:07.439440 3098 external_mini_cluster.cc:1428] Setting key a570f09485d5863f316426a7a8a53587
I20260925 05:11:07.439716 13513 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40031
I20260925 05:11:07.439882 13513 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:07.440106 13513 heartbeater.cc:511] Master 127.3.6.190:40031 requested a full tablet report, sending...
I20260925 05:11:07.440706 13322 ts_manager.cc:194] Registered new tserver with Master: 2bedfcfbb465476b9a62a531d415978b (127.3.6.129:39117)
I20260925 05:11:07.441326 13322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:38821
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:11:07.468030 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40031
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:07.554632 13518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:07.554878 13518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:07.554904 13518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:07.556115 13518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:07.556164 13518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:07.556191 13518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:11:07.557614 13518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:07.557818 13518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:07.558014 13518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:07.563192 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:07.563340 13518 server_base.cc:1034] running on GCE node
W20260925 05:11:07.563194 13523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:07.563443 13524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:07.563699 13518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:07.563982 13518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:07.565168 13518 hybrid_clock.cc:648] HybridClock initialized: now 1790313067565157 us; error 38 us; skew 500 ppm
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:07.567988 13518 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:11:07.568421 13518 webserver.cc:460] Webserver started at http://127.3.6.130:32881/ using document root <none> and password file <none>
I20260925 05:11:07.568625 13518 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:07.568671 13518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:07.568770 13518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:07.569847 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "6aff95339316490895ad21839af3fb86"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "ae1751f9941576256f36470d314c9ae6"
server_key_iv: "4e214dfad64d2d8166abb4c30e3cdf91"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.570161 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "6aff95339316490895ad21839af3fb86"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "ae1751f9941576256f36470d314c9ae6"
server_key_iv: "4e214dfad64d2d8166abb4c30e3cdf91"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.571498 13518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:11:07.572358 13533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.572541 13518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:11:07.572662 13518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "6aff95339316490895ad21839af3fb86"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "ae1751f9941576256f36470d314c9ae6"
server_key_iv: "4e214dfad64d2d8166abb4c30e3cdf91"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.572732 13518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:07.586563 13518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:07.588696 13518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:07.588819 13518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:07.589012 13518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:07.589346 13518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:07.589382 13518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.589406 13518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:07.589453 13518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.595423 13518 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:35057
I20260925 05:11:07.595503 13646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:35057 every 8 connection(s)
I20260925 05:11:07.595870 13518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:11:07.603979 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 13518
I20260925 05:11:07.604071 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:07.604282 3098 external_mini_cluster.cc:1428] Setting key 843d7bd3be3f5c0f451c6d271b66b0cc
I20260925 05:11:07.605727 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.596704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:38181 (local address 127.3.6.190:40031)
0925 05:11:07.596772 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.596774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.597006 (+ 232us) server_negotiation.cc:407] Connection header received
0925 05:11:07.597502 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.597503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.597541 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.597599 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:07.598552 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.599054 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.600200 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.600392 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.600460 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.602831 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:07.602832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:07.602833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:07.602840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:07.604258 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.604705 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.604705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.604706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.604771 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.605044 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.605045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.605045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.605173 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:07.605230 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.605587 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.605660 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:11:07.606068 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.596682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40031 (local address 127.3.6.130:38181)
0925 05:11:07.596867 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.596911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.597406 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.597663 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.597666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.598255 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:07.598460 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.598465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.599147 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.599147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.600109 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.600112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.600478 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.600479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.600613 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.601223 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:07.601232 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:07.602727 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:07.604361 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:07.604370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:07.604373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:07.604646 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:07.604839 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:07.604840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:07.604840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:07.604962 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:07.605244 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:07.605252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:07.605441 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.605566 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.605769 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260925 05:11:07.606664 13647 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40031
I20260925 05:11:07.606768 13647 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:07.607017 13647 heartbeater.cc:511] Master 127.3.6.190:40031 requested a full tablet report, sending...
I20260925 05:11:07.607474 13322 ts_manager.cc:194] Registered new tserver with Master: 6aff95339316490895ad21839af3fb86 (127.3.6.130:35057)
I20260925 05:11:07.608042 13322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:38181
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:11:07.636444 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40031
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:07.721299 13652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:07.721565 13652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:07.721611 13652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:07.722842 13652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:07.722925 13652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:07.722976 13652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:11:07.724583 13652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:07.724869 13652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:07.725121 13652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:07.730417 13658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:07.730449 13660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:07.730417 13657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:07.730705 13652 server_base.cc:1034] running on GCE node
I20260925 05:11:07.730960 13652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:07.731235 13652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:07.732388 13652 hybrid_clock.cc:648] HybridClock initialized: now 1790313067732365 us; error 37 us; skew 500 ppm
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:07.735177 13652 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:11:07.735800 13652 webserver.cc:460] Webserver started at http://127.3.6.131:46625/ using document root <none> and password file <none>
I20260925 05:11:07.736060 13652 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:07.736131 13652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:07.736292 13652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:11:07.737417 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "29c9c80ae64947c6b8a3876463df40bb"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "49962e3a89001888e05de5f5fcdec472"
server_key_iv: "82a5ac3bf65702fe3b0b7d3a75fa8af9"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.737810 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "29c9c80ae64947c6b8a3876463df40bb"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "49962e3a89001888e05de5f5fcdec472"
server_key_iv: "82a5ac3bf65702fe3b0b7d3a75fa8af9"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.739311 13652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:11:07.740240 13667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.740435 13652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:11:07.740569 13652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "29c9c80ae64947c6b8a3876463df40bb"
format_stamp: "Formatted at 2026-09-25 05:11:07 on dist-test-slave-1lvv"
server_key: "49962e3a89001888e05de5f5fcdec472"
server_key_iv: "82a5ac3bf65702fe3b0b7d3a75fa8af9"
server_key_version: "encryptionkey@0"
I20260925 05:11:07.740659 13652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:07.775269 13652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:07.777901 13652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:07.778079 13652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:07.778349 13652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:07.778685 13652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:07.778739 13652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.778775 13652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:07.778822 13652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:07.785298 13652 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:36941
I20260925 05:11:07.785341 13780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:36941 every 8 connection(s)
I20260925 05:11:07.785713 13652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:11:07.792896 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 13652
I20260925 05:11:07.793016 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:07.793321 3098 external_mini_cluster.cc:1428] Setting key 63bc0410a32a32a2ca77cfdfd6f4ee58
I20260925 05:11:07.795900 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.786552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:58847 (local address 127.3.6.190:40031)
0925 05:11:07.786628 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.786630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.786812 (+ 182us) server_negotiation.cc:407] Connection header received
0925 05:11:07.787349 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.787351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.787362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.787433 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:07.788578 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.789110 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.790122 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.790313 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.790376 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.792721 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:07.792723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:07.792723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:07.792732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:07.794564 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.795004 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.795004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.795005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.795058 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.795282 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.795283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.795283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.795391 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:07.795442 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.795693 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.795806 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:11:07.796088 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.786505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:40031 (local address 127.3.6.131:58847)
0925 05:11:07.786671 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.786716 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.787254 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.787494 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.787497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.788224 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:07.788433 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.788439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.789207 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.789208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.790029 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.790031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.790396 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.790397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.790504 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.791063 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:07.791072 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:07.792579 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:07.794649 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:07.794658 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:07.794661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:07.794934 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:07.795117 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:07.795117 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:07.795118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:07.795224 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:07.795448 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:07.795452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:07.795578 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.795666 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.795859 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260925 05:11:07.796641 13781 heartbeater.cc:347] Connected to a master server at 127.3.6.190:40031
I20260925 05:11:07.796749 13781 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:07.796941 13781 heartbeater.cc:511] Master 127.3.6.190:40031 requested a full tablet report, sending...
I20260925 05:11:07.797346 13322 ts_manager.cc:194] Registered new tserver with Master: 29c9c80ae64947c6b8a3876463df40bb (127.3.6.131:36941)
I20260925 05:11:07.797849 13322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:58847
I20260925 05:11:07.805374 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:11:07.812922 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.806681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.3.6.190:40031)
0925 05:11:07.806758 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.806760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.806819 (+ 59us) server_negotiation.cc:407] Connection header received
0925 05:11:07.806877 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.806878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.806887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.806935 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:07.807499 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.808064 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.808780 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.808957 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.809008 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.810222 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:07.810223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:07.810223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:07.810228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:07.811599 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.812023 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.812024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.812024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.812120 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:07.812319 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:07.812319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:07.812320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:07.812466 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:07.812546 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.812718 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.812796 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:11:07.814739 13322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:11:07.816654 13322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:11:07.823671 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.819371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:39117 (local address 127.0.0.1:60838)
0925 05:11:07.819738 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.819750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.819827 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.820284 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.820284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.820287 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:07.820347 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.820349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.821628 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.821629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.822660 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.822662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.823465 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.823465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.823523 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.823525 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.823570 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.823598 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":83,"threads_started":1}
I20260925 05:11:07.823911 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.819471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.3.6.130:35057)
0925 05:11:07.819780 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.819786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.819797 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:11:07.819841 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.819844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.819860 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.819952 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:07.820176 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.821144 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.822542 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.823248 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.823334 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.823396 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.823640 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.823683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
I20260925 05:11:07.823717 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.819685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.3.6.129:39117)
0925 05:11:07.820141 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.820146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.820156 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:11:07.820194 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.820196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.820212 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.820287 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:07.820413 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.821515 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.822750 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.823389 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.823464 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.823518 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.823595 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.823633 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":102,"threads_started":1}
I20260925 05:11:07.823939 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.819335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:35057 (local address 127.0.0.1:47300)
0925 05:11:07.819575 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.819619 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.819691 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.819968 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.819972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.819975 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:07.820075 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.820078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.821244 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.821246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.822452 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.822454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.823332 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.823333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.823412 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.823426 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.823495 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.823553 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260925 05:11:07.824158 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.819603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:36941 (local address 127.0.0.1:48348)
0925 05:11:07.820382 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.820392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.820446 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.820629 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.820630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.820632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:07.820682 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.820684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.821840 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.821842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.822954 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.822957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.823940 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.823941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.824001 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.824005 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.824054 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.824091 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":81,"threads_started":1}
I20260925 05:11:07.824456 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.819889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.3.6.131:36941)
0925 05:11:07.820109 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.820114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.820475 (+ 361us) server_negotiation.cc:407] Connection header received
0925 05:11:07.820523 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.820526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.820541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.820621 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:07.820743 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.821747 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.823045 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.823789 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.823854 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.823907 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.824331 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.824378 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":120,"threads_started":1}
I20260925 05:11:07.824858 13447 tablet_service.cc:1467] Processing CreateTablet for tablet 107a840523384b02bf838efa98811432 (DEFAULT_TABLE table=test-table [id=55db48eab5604041a94d2c5641caaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:07.824857 13581 tablet_service.cc:1467] Processing CreateTablet for tablet 107a840523384b02bf838efa98811432 (DEFAULT_TABLE table=test-table [id=55db48eab5604041a94d2c5641caaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:07.825157 13447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107a840523384b02bf838efa98811432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:07.825157 13581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107a840523384b02bf838efa98811432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:07.827803 13798 tablet_bootstrap.cc:492] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Bootstrap starting.
I20260925 05:11:07.827966 13799 tablet_bootstrap.cc:492] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b: Bootstrap starting.
I20260925 05:11:07.828102 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 107a840523384b02bf838efa98811432 (DEFAULT_TABLE table=test-table [id=55db48eab5604041a94d2c5641caaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:07.828477 13798 tablet_bootstrap.cc:654] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:07.828470 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107a840523384b02bf838efa98811432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:07.828663 13799 tablet_bootstrap.cc:654] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:07.829000 13798 log.cc:824] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:07.829210 13799 log.cc:824] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:07.829810 13798 tablet_bootstrap.cc:492] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: No bootstrap required, opened a new log
I20260925 05:11:07.829898 13798 ts_tablet_manager.cc:1397] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:11:07.830019 13799 tablet_bootstrap.cc:492] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b: No bootstrap required, opened a new log
I20260925 05:11:07.830096 13799 ts_tablet_manager.cc:1397] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:11:07.831621 13798 raft_consensus.cc:348] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.831810 13798 raft_consensus.cc:374] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:07.831869 13798 raft_consensus.cc:729] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aff95339316490895ad21839af3fb86, State: Initialized, Role: FOLLOWER
I20260925 05:11:07.832028 13802 tablet_bootstrap.cc:492] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Bootstrap starting.
I20260925 05:11:07.832041 13798 consensus_queue.cc:260] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.832342 13798 ts_tablet_manager.cc:1428] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 05:11:07.832271 13799 raft_consensus.cc:348] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.832442 13799 raft_consensus.cc:374] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:07.832481 13647 heartbeater.cc:503] Master 127.3.6.190:40031 was elected leader, sending a full tablet report...
I20260925 05:11:07.832492 13799 raft_consensus.cc:729] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bedfcfbb465476b9a62a531d415978b, State: Initialized, Role: FOLLOWER
I20260925 05:11:07.832659 13799 consensus_queue.cc:260] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.833005 13802 tablet_bootstrap.cc:654] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:07.832983 13513 heartbeater.cc:503] Master 127.3.6.190:40031 was elected leader, sending a full tablet report...
I20260925 05:11:07.832979 13799 ts_tablet_manager.cc:1428] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:07.833726 13802 log.cc:824] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:07.834575 13802 tablet_bootstrap.cc:492] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: No bootstrap required, opened a new log
I20260925 05:11:07.834640 13802 ts_tablet_manager.cc:1397] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:11:07.836326 13802 raft_consensus.cc:348] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.836485 13802 raft_consensus.cc:374] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:07.836508 13802 raft_consensus.cc:729] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c9c80ae64947c6b8a3876463df40bb, State: Initialized, Role: FOLLOWER
I20260925 05:11:07.836630 13802 consensus_queue.cc:260] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.836895 13802 ts_tablet_manager.cc:1428] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:11:07.836966 13781 heartbeater.cc:503] Master 127.3.6.190:40031 was elected leader, sending a full tablet report...
W20260925 05:11:07.846447 13648 tablet.cc:2369] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:07.850950 13804 raft_consensus.cc:482] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:11:07.851116 13804 raft_consensus.cc:504] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.851523 13804 leader_election.cc:288] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aff95339316490895ad21839af3fb86 (127.3.6.130:35057), 29c9c80ae64947c6b8a3876463df40bb (127.3.6.131:36941)
I20260925 05:11:07.854581 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.851901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:42225 (local address 127.3.6.130:35057)
0925 05:11:07.851975 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.851977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.851987 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:11:07.852020 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.852021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.852034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.852121 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:07.852313 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.852862 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.853815 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.854216 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.854271 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.854280 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.854463 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.854507 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:11:07.854646 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.851804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:35057 (local address 127.3.6.129:42225)
0925 05:11:07.851850 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.851864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.851913 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.852138 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.852139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.852141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:07.852210 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.852212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.852941 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.852942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.853724 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.853726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.854296 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.854297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.854386 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.854390 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.854447 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.854478 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 05:11:07.854672 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.852063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:49853 (local address 127.3.6.131:36941)
0925 05:11:07.852116 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.852119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.852248 (+ 129us) server_negotiation.cc:407] Connection header received
0925 05:11:07.852280 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.852281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.852295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.852347 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:07.852489 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.852956 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.853893 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.854291 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.854364 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.854373 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.854554 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.854601 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:11:07.854741 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.851936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:36941 (local address 127.3.6.129:49853)
0925 05:11:07.852169 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.852179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:07.852238 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:07.852365 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:07.852366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:07.852367 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:07.852429 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.852430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.853046 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.853047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.853805 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:07.853807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.854388 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.854389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:07.854473 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.854477 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:11:07.854534 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:11:07.854571 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":142,"threads_started":1}
I20260925 05:11:07.855070 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107a840523384b02bf838efa98811432" candidate_uuid: "2bedfcfbb465476b9a62a531d415978b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aff95339316490895ad21839af3fb86" is_pre_election: true
I20260925 05:11:07.855105 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107a840523384b02bf838efa98811432" candidate_uuid: "2bedfcfbb465476b9a62a531d415978b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c9c80ae64947c6b8a3876463df40bb" is_pre_election: true
I20260925 05:11:07.855262 13601 raft_consensus.cc:2454] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bedfcfbb465476b9a62a531d415978b in term 0.
I20260925 05:11:07.855262 13735 raft_consensus.cc:2454] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bedfcfbb465476b9a62a531d415978b in term 0.
I20260925 05:11:07.855594 13402 leader_election.cc:302] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c9c80ae64947c6b8a3876463df40bb, 2bedfcfbb465476b9a62a531d415978b; no voters:
I20260925 05:11:07.855749 13804 raft_consensus.cc:2781] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:11:07.855861 13804 raft_consensus.cc:482] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:11:07.855902 13804 raft_consensus.cc:3037] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:07.856665 13804 raft_consensus.cc:504] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.856805 13804 leader_election.cc:288] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [CANDIDATE]: Term 1 election: Requested vote from peers 6aff95339316490895ad21839af3fb86 (127.3.6.130:35057), 29c9c80ae64947c6b8a3876463df40bb (127.3.6.131:36941)
I20260925 05:11:07.857002 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107a840523384b02bf838efa98811432" candidate_uuid: "2bedfcfbb465476b9a62a531d415978b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c9c80ae64947c6b8a3876463df40bb"
I20260925 05:11:07.857031 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107a840523384b02bf838efa98811432" candidate_uuid: "2bedfcfbb465476b9a62a531d415978b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aff95339316490895ad21839af3fb86"
I20260925 05:11:07.857098 13735 raft_consensus.cc:3037] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:07.857118 13601 raft_consensus.cc:3037] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:07.857841 13601 raft_consensus.cc:2454] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bedfcfbb465476b9a62a531d415978b in term 1.
I20260925 05:11:07.857858 13735 raft_consensus.cc:2454] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bedfcfbb465476b9a62a531d415978b in term 1.
I20260925 05:11:07.858022 13402 leader_election.cc:302] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bedfcfbb465476b9a62a531d415978b, 6aff95339316490895ad21839af3fb86; no voters:
I20260925 05:11:07.858173 13804 raft_consensus.cc:2781] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:07.858322 13804 raft_consensus.cc:686] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 1 LEADER]: Becoming Leader. State: Replica: 2bedfcfbb465476b9a62a531d415978b, State: Running, Role: LEADER
I20260925 05:11:07.858428 13804 consensus_queue.cc:237] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } }
I20260925 05:11:07.859324 13322 catalog_manager.cc:5495] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bedfcfbb465476b9a62a531d415978b (127.3.6.129). New cstate: current_term: 1 leader_uuid: "2bedfcfbb465476b9a62a531d415978b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bedfcfbb465476b9a62a531d415978b" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 39117 } health_report { overall_health: HEALTHY } } }
I20260925 05:11:07.869908 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.867452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.3.6.129:39117)
0925 05:11:07.867523 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:07.867527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:07.867543 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:11:07.867580 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:07.867581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:07.867600 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:07.867666 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:07.867845 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.868313 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.869157 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:07.869340 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:07.869393 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:07.869542 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:07.869647 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:11:07.869713 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:07.869760 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 05:11:07.869778 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:11:07.871928 13735 raft_consensus.cc:1261] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 1 FOLLOWER]: Refusing update from remote peer 2bedfcfbb465476b9a62a531d415978b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:07.871984 13601 raft_consensus.cc:1261] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 1 FOLLOWER]: Refusing update from remote peer 2bedfcfbb465476b9a62a531d415978b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:07.872368 13804 consensus_queue.cc:1035] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c9c80ae64947c6b8a3876463df40bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:07.872536 13808 consensus_queue.cc:1035] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aff95339316490895ad21839af3fb86" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:07.876139 13812 mvcc.cc:205] Tried to move back new op lower bound from 7333122326000160768 to 7333122325948870656. Current Snapshot: MvccSnapshot[applied={T|T < 7333122326000160768}]
I20260925 05:11:07.876369 13813 mvcc.cc:205] Tried to move back new op lower bound from 7333122326000160768 to 7333122325948870656. Current Snapshot: MvccSnapshot[applied={T|T < 7333122326000160768}]
I20260925 05:11:07.877651 13322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:07.877764 13322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:07.878583 13322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f42ca3893e2a4b59a6d908d6f089ae2f: Sending DeleteTablet for 3 replicas of tablet 107a840523384b02bf838efa98811432
I20260925 05:11:07.878953 13447 tablet_service.cc:1514] Processing DeleteTablet for tablet 107a840523384b02bf838efa98811432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:07 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:60838
I20260925 05:11:07.879004 13581 tablet_service.cc:1514] Processing DeleteTablet for tablet 107a840523384b02bf838efa98811432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:07 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:47300
I20260925 05:11:07.879212 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet 107a840523384b02bf838efa98811432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:07 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:48348
I20260925 05:11:07.879227 13820 tablet_replica.cc:330] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b: stopping tablet replica
I20260925 05:11:07.879328 13820 raft_consensus.cc:2229] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:11:07.879403 13822 tablet_replica.cc:330] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: stopping tablet replica
I20260925 05:11:07.879454 13820 raft_consensus.cc:2258] T 107a840523384b02bf838efa98811432 P 2bedfcfbb465476b9a62a531d415978b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:07.879496 13822 raft_consensus.cc:2229] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:07.879596 13822 raft_consensus.cc:2258] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:07.879858 13821 tablet_replica.cc:330] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: stopping tablet replica
I20260925 05:11:07.879979 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 13384
I20260925 05:11:07.879998 13821 raft_consensus.cc:2229] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:07.880100 13822 ts_tablet_manager.cc:1905] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:07.880230 13821 raft_consensus.cc:2258] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:07.880944 13821 ts_tablet_manager.cc:1905] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:07.882259 13822 ts_tablet_manager.cc:1918] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:07.882359 13822 log.cc:1196] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/107a840523384b02bf838efa98811432
I20260925 05:11:07.882678 13822 ts_tablet_manager.cc:1939] T 107a840523384b02bf838efa98811432 P 29c9c80ae64947c6b8a3876463df40bb: Deleting consensus metadata
I20260925 05:11:07.882858 13821 ts_tablet_manager.cc:1918] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:07.882936 13821 log.cc:1196] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/107a840523384b02bf838efa98811432
I20260925 05:11:07.883260 13821 ts_tablet_manager.cc:1939] T 107a840523384b02bf838efa98811432 P 6aff95339316490895ad21839af3fb86: Deleting consensus metadata
I20260925 05:11:07.883253 13308 catalog_manager.cc:4841] TS 29c9c80ae64947c6b8a3876463df40bb (127.3.6.131:36941): tablet 107a840523384b02bf838efa98811432 (table test-table [id=55db48eab5604041a94d2c5641caaa23]) successfully deleted
I20260925 05:11:07.883689 13308 catalog_manager.cc:4841] TS 6aff95339316490895ad21839af3fb86 (127.3.6.130:35057): tablet 107a840523384b02bf838efa98811432 (table test-table [id=55db48eab5604041a94d2c5641caaa23]) successfully deleted
W20260925 05:11:07.887820 13306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:39117 (error 108)
I20260925 05:11:07.888039 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 13518
I20260925 05:11:07.889021 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:07.888862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:39117 (local address 127.0.0.1:60854)
0925 05:11:07.888930 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:07.888977 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:39117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 05:11:07.889154 13306 catalog_manager.cc:4584] TS 2bedfcfbb465476b9a62a531d415978b (127.3.6.129:39117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 107a840523384b02bf838efa98811432: Network error: Client connection negotiation failed: client connection to 127.3.6.129:39117: connect: Connection refused (error 111)
I20260925 05:11:07.893122 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 13652
I20260925 05:11:07.897804 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 13290
2026-09-25T05:11:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1974 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13826](info): set up 2 sockets
Sep 25 05:11:07 dist-test-slave-1lvv krb5kdc[13826](info): commencing operation
krb5kdc: starting...
W20260925 05:11:08.941098 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:11:09 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/rangeradmin_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/rangeradmin_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/rangerlookup_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/rangerlookup_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
I20260925 05:11:09.131208 3098 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskH05q1Z/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/postgres -l logfile start
2026-09-25 05:11:11.904 UTC [13856] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:11:11.904 UTC [13856] LOG: listening on IPv4 address "127.3.6.148", port 36507
2026-09-25 05:11:11.909 UTC [13856] LOG: listening on Unix socket "/tmp/.s.PGSQL.36507"
2026-09-25 05:11:11.946 UTC [13859] LOG: database system was shut down at 2026-09-25 05:11:10 UTC
2026-09-25 05:11:11.951 UTC [13856] LOG: database system is ready to accept connections
I20260925 05:11:12.900717 3098 mini_postgres.cc:96] Postgres bound to 36507
2026-09-25 05:11:12.907 UTC [13867] FATAL: database "slave" does not exist
127.3.6.148:36507 - accepting connections
I20260925 05:11:12.908546 3098 mini_ranger.cc:161] Starting Ranger...
I20260925 05:11:12.924425 3098 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:11:13.486105 3098 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:11:13.486203 3098 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-admin
2026-09-25 05:11:13,849 [I] DB FLAVOR :POSTGRES
2026-09-25 05:11:13,849 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:11:13,849 [I] Checking connection..
2026-09-25 05:11:13,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:11:14,126 [I] Checking connection passed.
2026-09-25 05:11:14,126 [I] --------- Verifying version history table ---------
2026-09-25 05:11:14,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:11:14,393 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:11:14,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:11:14,634 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:11:14,634 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:11:14,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:11:14.907 UTC [13931] WARNING: there is no transaction in progress
2026-09-25 05:11:14,913 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:11:14,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:11:15,156 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:11:15,157 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:11:15,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:11:15,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:11:15,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1lvv.c.gcp-upstream.internal','N') ;"
2026-09-25 05:11:15,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:11:16,128 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:11:16,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:11:16,392 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:11:16,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:11:16,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:11:16,886 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:11:16,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:11:17.947 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:17.966 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:17.985 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.002 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.023 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.042 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.105 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.113 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.130 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.141 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.153 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.169 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.180 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.188 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.195 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18.203 UTC [14039] WARNING: there is no transaction in progress
2026-09-25 05:11:18,944 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:11:18,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:11:19,180 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:11:19,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:11:19,413 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:11:19,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:11:19,694 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:11:19,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:11:19,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:11:20,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1lvv.c.gcp-upstream.internal';"
2026-09-25 05:11:20,459 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:11:20,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:11:20,705 [I] DB_PATCHES have already been applied
I20260925 05:11:20.711444 3098 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-admin:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskH05q1Z/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:11:20.711521 3098 mini_ranger.cc:193] Using host: 127.3.6.148
I20260925 05:11:20.715216 3098 mini_ranger.cc:227] Ranger admin URL: http://127.3.6.148:54195
Sep 25, 2026 5:11:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:11:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:11:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:11:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:11:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:11:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:11:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:11:21.869899 3098 mini_ranger.cc:160] Time spent starting Ranger: real 8.961s user 0.000s sys 0.012s
Sep 25, 2026 5:11:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:11:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 5:11:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:11:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:11:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:11:44.648480 3098 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/rangerkms_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/rangerkms_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.6.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.6.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:11:44 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:11:44.745393 3098 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:11:44.755883 3098 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:11:45.675225 3098 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:11:45.675297 3098 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms
I20260925 05:11:45.675309 3098 mini_ranger_kms.cc:227] Using postgres at 127.3.6.148:36507
2026-09-25 05:11:45,950 [I] DB FLAVOR :POSTGRES
2026-09-25 05:11:45,950 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:11:45,950 [I] Checking connection
2026-09-25 05:11:45,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:11:46,219 [I] connection success
2026-09-25 05:11:46,220 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:11:46,220 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:11:46,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:11:46,491 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:11:46,491 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:11:46,492 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:11:46,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:36507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:11:46,776 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:11:47.252760 3098 mini_ranger_kms.cc:310] Created kms service
I20260925 05:11:47.440599 3098 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:11:47.488277 3098 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:11:47 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313104, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:11:47.949932 3098 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:11:47.950079 3098 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms:/tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:11:47.950101 3098 mini_ranger_kms.cc:240] Using host: 127.3.6.148
I20260925 05:11:47.954035 3098 mini_ranger_kms.cc:276] Ranger KMS PID: 14220
I20260925 05:11:47.954173 3098 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.6.148:54999
Sep 25, 2026 5:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 05:11:50 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790313110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.6.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:50.296540 3098 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.551s user 0.001s sys 0.257s
I20260925 05:11:50.296634 3098 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:11:50.296725 3098 mini_ranger_kms.cc:401] 127.3.6.148:54999/kms/v1/keys
Sep 25 05:11:50 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.6.148@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
Sep 25 05:11:51 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313110, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.6.148@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
Sep 25 05:11:53 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:11:53.425422 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45885
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:45885
--ranger_config_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:54999/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:53.508564 14270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:53.508834 14270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:53.508876 14270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:53.510032 14270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:11:53.510118 14270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:53.510143 14270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:53.510165 14270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:11:53.510193 14270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:11:53.511797 14270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.6.148:54999/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:45885
--ranger_config_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:53.512089 14270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:53.512320 14270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:53.517823 14278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:53.517863 14276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:53.517861 14270 server_base.cc:1034] running on GCE node
W20260925 05:11:53.517843 14275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:53.518371 14270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:53.518714 14270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:53.519825 14270 hybrid_clock.cc:648] HybridClock initialized: now 1790313113519812 us; error 33 us; skew 500 ppm
Sep 25 05:11:53 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:53.522351 14270 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:11:53.522842 14270 webserver.cc:460] Webserver started at http://127.3.6.190:39075/ using document root <none> and password file <none>
I20260925 05:11:53.523092 14270 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:53.523155 14270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:53.523320 14270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:11:53 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:11:53.577303 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "52ff43ab219546a1a35bcd5ff989fac1"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-1lvv"
server_key: "583d372d62cf5715f075c9d2c8937301"
server_key_iv: "05878ea4160636c04bab166f7693fd10"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:53.577709 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "52ff43ab219546a1a35bcd5ff989fac1"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-1lvv"
server_key: "583d372d62cf5715f075c9d2c8937301"
server_key_iv: "05878ea4160636c04bab166f7693fd10"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:53.579416 14270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:53.594877 14285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:53.595155 14270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:53.595286 14270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "52ff43ab219546a1a35bcd5ff989fac1"
format_stamp: "Formatted at 2026-09-25 05:11:53 on dist-test-slave-1lvv"
server_key: "583d372d62cf5715f075c9d2c8937301"
server_key_iv: "05878ea4160636c04bab166f7693fd10"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:53.595393 14270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:53.623934 14270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:53.625976 14270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:53.626155 14270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:53.630707 14270 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:45885
I20260925 05:11:53.630767 14337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:45885 every 8 connection(s)
I20260925 05:11:53.631279 14270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:11:53.632284 14338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:53.632630 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 14270
I20260925 05:11:53.632735 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:11:53.632983 3098 external_mini_cluster.cc:1428] Setting key 72171d0748e57d3fda5fe3f8e2b9592b
I20260925 05:11:53.636072 14338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1: Bootstrap starting.
I20260925 05:11:53.636883 14338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:53.637229 14338 log.cc:824] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:53.638080 14338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1: No bootstrap required, opened a new log
I20260925 05:11:53.640647 14338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ff43ab219546a1a35bcd5ff989fac1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45885 } }
I20260925 05:11:53.640852 14338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:53.640892 14338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ff43ab219546a1a35bcd5ff989fac1, State: Initialized, Role: FOLLOWER
I20260925 05:11:53.641074 14338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ff43ab219546a1a35bcd5ff989fac1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45885 } }
I20260925 05:11:53.641160 14338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:11:53.641204 14338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:11:53.641253 14338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 05:11:53 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:11:53.641891 14338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ff43ab219546a1a35bcd5ff989fac1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45885 } }
I20260925 05:11:53.642014 14338 leader_election.cc:302] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ff43ab219546a1a35bcd5ff989fac1; no voters:
I20260925 05:11:53.642197 14338 leader_election.cc:288] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:11:53.642246 14343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:53.642433 14343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [term 1 LEADER]: Becoming Leader. State: Replica: 52ff43ab219546a1a35bcd5ff989fac1, State: Running, Role: LEADER
I20260925 05:11:53.642493 14338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:11:53.642518 14343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ff43ab219546a1a35bcd5ff989fac1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45885 } }
I20260925 05:11:53.642925 14344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52ff43ab219546a1a35bcd5ff989fac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ff43ab219546a1a35bcd5ff989fac1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45885 } } }
I20260925 05:11:53.643191 14344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:53.643110 14345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ff43ab219546a1a35bcd5ff989fac1. Latest consensus state: current_term: 1 leader_uuid: "52ff43ab219546a1a35bcd5ff989fac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ff43ab219546a1a35bcd5ff989fac1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45885 } } }
I20260925 05:11:53.643363 14345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:11:53.643517 14352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:11:53.644099 14352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:11:53.644378 14338 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:11:53.646219 14352 catalog_manager.cc:1347] Generated new cluster ID: e8f24ba010eb4e14b5198746a69dc67b
I20260925 05:11:53.646266 14352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:11:53.646485 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:53.633733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.3.6.190:45885)
0925 05:11:53.633914 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:53.633924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:53.633956 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:11:53.634664 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:53.634674 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:53.634746 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:53.634970 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:53.635780 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.637171 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.638402 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:53.638991 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:53.639086 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:53.641962 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:53.641966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:53.641966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:53.641978 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:53.644750 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:53.645178 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:53.645182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:53.645183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:53.645280 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:53.645539 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:53.645539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:53.645540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:53.645861 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:53.646025 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:53.646153 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:11:53.646270 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260925 05:11:53.663957 14352 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:11:53.664772 14352 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:11:53.668033 14352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1: Generated new TSK 0
I20260925 05:11:53.668323 14352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:11:54.383000 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:54999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45885
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:54.481115 14383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:54.481369 14383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:54.481426 14383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:54.482828 14383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:54.482908 14383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:54.482956 14383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:11:54.484567 14383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.6.148:54999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:54.484843 14383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:54.485085 14383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:54.490516 14391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:54.490672 14389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:54.490516 14388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:54.491231 14383 server_base.cc:1034] running on GCE node
I20260925 05:11:54.491508 14383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:54.491807 14383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:54.492966 14383 hybrid_clock.cc:648] HybridClock initialized: now 1790313114492957 us; error 33 us; skew 500 ppm
Sep 25 05:11:54 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:54.495589 14383 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:11:54.496085 14383 webserver.cc:460] Webserver started at http://127.3.6.129:38037/ using document root <none> and password file <none>
I20260925 05:11:54.496345 14383 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:54.496414 14383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:54.496567 14383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:11:54 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:11:54.511505 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "109a3932c94b4645bc38d8dc4766164a"
format_stamp: "Formatted at 2026-09-25 05:11:54 on dist-test-slave-1lvv"
server_key: "b6254ab093248178401fc9492f301ac1"
server_key_iv: "b4a15a97e98997135e1066f9e6ec093d"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:54.512097 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "109a3932c94b4645bc38d8dc4766164a"
format_stamp: "Formatted at 2026-09-25 05:11:54 on dist-test-slave-1lvv"
server_key: "b6254ab093248178401fc9492f301ac1"
server_key_iv: "b4a15a97e98997135e1066f9e6ec093d"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:54.515662 14383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:11:54.529150 14398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.529417 14383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:11:54.529626 14383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "109a3932c94b4645bc38d8dc4766164a"
format_stamp: "Formatted at 2026-09-25 05:11:54 on dist-test-slave-1lvv"
server_key: "b6254ab093248178401fc9492f301ac1"
server_key_iv: "b4a15a97e98997135e1066f9e6ec093d"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:54.529801 14383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:54.556957 14383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:54.559356 14383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:54.559514 14383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:54.559733 14383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:54.560173 14383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:54.560225 14383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.560254 14383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:54.560307 14383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.567948 14383 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:40691
I20260925 05:11:54.568089 14513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:40691 every 8 connection(s)
I20260925 05:11:54.568542 14383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:11:54.571050 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 14383
I20260925 05:11:54.571158 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:11:54.571367 3098 external_mini_cluster.cc:1428] Setting key 9c0f609ab90eab526a35e363051a30eb
Sep 25 05:11:54 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
I20260925 05:11:54.581863 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.569928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:54775 (local address 127.3.6.190:45885)
0925 05:11:54.570099 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.570102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.570266 (+ 164us) server_negotiation.cc:407] Connection header received
0925 05:11:54.570819 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.570821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.570835 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.570912 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:54.571747 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.572552 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.574134 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.574462 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.574524 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.577876 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:54.577878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:54.577878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:54.577886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:54.579966 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.580531 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.580532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.580533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.580613 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.580974 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.580974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.580975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.581133 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:54.581215 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.581634 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.581735 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260925 05:11:54.582031 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.569676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45885 (local address 127.3.6.129:54775)
0925 05:11:54.570122 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.570168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.570708 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.571070 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.571075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.571341 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:54.571643 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.571650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.572654 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.572655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.574024 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.574028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.574598 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.574600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.574754 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.575899 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:54.575910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:54.577756 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:54.580068 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:54.580075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:54.580077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:54.580396 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:54.580714 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:54.580715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:54.580716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:54.580879 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:54.581228 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:54.581234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:54.581455 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.581581 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.581756 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":103,"threads_started":1}
I20260925 05:11:54.582650 14514 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45885
I20260925 05:11:54.582751 14514 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:54.582957 14514 heartbeater.cc:511] Master 127.3.6.190:45885 requested a full tablet report, sending...
I20260925 05:11:54.583743 14302 ts_manager.cc:194] Registered new tserver with Master: 109a3932c94b4645bc38d8dc4766164a (127.3.6.129:40691)
I20260925 05:11:54.584693 14302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:54775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:11:54.606933 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:54999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45885
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:54.724800 14520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:54.725184 14520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:54.725286 14520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:54.727208 14520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:54.727329 14520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:54.727415 14520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:11:54.729779 14520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.6.148:54999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:54.730347 14520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:54.730691 14520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:54.737339 14530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:54.737483 14528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:54.737367 14527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:54.738520 14520 server_base.cc:1034] running on GCE node
I20260925 05:11:54.738756 14520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:54.739069 14520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 25 05:11:54 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:54.740262 14520 hybrid_clock.cc:648] HybridClock initialized: now 1790313114740243 us; error 29 us; skew 500 ppm
Sep 25 05:11:54 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:54.743602 14520 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:11:54.744225 14520 webserver.cc:460] Webserver started at http://127.3.6.130:40709/ using document root <none> and password file <none>
I20260925 05:11:54.744625 14520 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:54.744705 14520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:54.744911 14520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:11:54 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:11:54.761672 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "818a36ba6df0463f9231b2b27ba66acf"
format_stamp: "Formatted at 2026-09-25 05:11:54 on dist-test-slave-1lvv"
server_key: "0209e7bfed575dbe0efa22a62a872b72"
server_key_iv: "847dc5fcdeca3db0e3c748031e94efd3"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:54.762152 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "818a36ba6df0463f9231b2b27ba66acf"
format_stamp: "Formatted at 2026-09-25 05:11:54 on dist-test-slave-1lvv"
server_key: "0209e7bfed575dbe0efa22a62a872b72"
server_key_iv: "847dc5fcdeca3db0e3c748031e94efd3"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:54.764240 14520 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 05:11:54.784018 14537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.784303 14520 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260925 05:11:54.784394 14520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "818a36ba6df0463f9231b2b27ba66acf"
format_stamp: "Formatted at 2026-09-25 05:11:54 on dist-test-slave-1lvv"
server_key: "0209e7bfed575dbe0efa22a62a872b72"
server_key_iv: "847dc5fcdeca3db0e3c748031e94efd3"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:54.784494 14520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:54.805505 14520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:54.808266 14520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:54.808501 14520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:54.808790 14520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:54.809269 14520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:54.812531 14520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260925 05:11:54.812657 14520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:54.813135 14520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:54.822674 14520 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:33239
I20260925 05:11:54.822968 14651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:33239 every 8 connection(s)
I20260925 05:11:54.823318 14520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:11:54.824558 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 14520
I20260925 05:11:54.824673 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:11:54.824893 3098 external_mini_cluster.cc:1428] Setting key 2823cd95c77d779424d0088c00ad0158
Sep 25 05:11:54 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
I20260925 05:11:54.842974 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.824430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:52987 (local address 127.3.6.190:45885)
0925 05:11:54.825095 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:54.825098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:54.825112 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:11:54.825538 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:54.825540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:54.825551 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:54.825581 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:54.826505 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.827289 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.828999 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.829297 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.829340 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.833886 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:54.833888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:54.833889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:54.833899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:54.835747 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.841637 (+ 5890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.841639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.841640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.841721 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:54.842095 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:54.842096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:54.842097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:54.842259 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:54.842315 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:54.842748 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 05:11:54.842869 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628}
I20260925 05:11:54.843823 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:54.824408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45885 (local address 127.3.6.130:52987)
0925 05:11:54.824625 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:54.824693 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:54.825420 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:54.825786 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:54.825791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:54.826090 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:54.826381 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.826388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.827419 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.827420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.828887 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:54.828890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:54.829439 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:54.829440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:54.829585 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:54.831805 (+ 2220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:54.831818 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:54.833715 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:54.835905 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:54.835918 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:54.835921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:54.836284 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:54.841814 (+ 5530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:54.841815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:54.841815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:54.842007 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:54.842351 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:54.842358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:54.842577 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 05:11:54.843346 (+ 769us) client_negotiation.cc:240] Negotiation successful
0925 05:11:54.843544 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20260925 05:11:54.844537 14652 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45885
I20260925 05:11:54.844636 14652 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:54.844835 14652 heartbeater.cc:511] Master 127.3.6.190:45885 requested a full tablet report, sending...
I20260925 05:11:54.845291 14302 ts_manager.cc:194] Registered new tserver with Master: 818a36ba6df0463f9231b2b27ba66acf (127.3.6.130:33239)
I20260925 05:11:54.845774 14302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:52987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:11:54.880157 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:54999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45885
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:11:54.992271 14657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:11:54.992635 14657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:11:54.992726 14657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:11:54.994504 14657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:11:54.994621 14657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:11:54.994714 14657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:11:54.997149 14657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.6.148:54999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:45885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:11:54.997630 14657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:11:54.997994 14657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:11:55.012159 14662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:55.012135 14665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:11:55.012205 14663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:11:55.013932 14657 server_base.cc:1034] running on GCE node
I20260925 05:11:55.014228 14657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:11:55.014571 14657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:11:55.015748 14657 hybrid_clock.cc:648] HybridClock initialized: now 1790313115015729 us; error 41 us; skew 500 ppm
Sep 25 05:11:55 dist-test-slave-1lvv krb5kdc[13826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313115, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:11:55.019445 14657 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:11:55.020147 14657 webserver.cc:460] Webserver started at http://127.3.6.131:46335/ using document root <none> and password file <none>
I20260925 05:11:55.020529 14657 fs_manager.cc:359] Metadata directory not provided
I20260925 05:11:55.020663 14657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:11:55.020869 14657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:11:55 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313115, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:11:55.045082 14657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "089ca630f03e4cf7b556735ad21d247a"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-1lvv"
server_key: "7aebcc4c4d917705da5a2335b0849092"
server_key_iv: "0aff41d2740301e093599d31e822db37"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:55.045624 14657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "089ca630f03e4cf7b556735ad21d247a"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-1lvv"
server_key: "7aebcc4c4d917705da5a2335b0849092"
server_key_iv: "0aff41d2740301e093599d31e822db37"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:55.048270 14657 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:11:55.063024 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:55.063318 14657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:11:55.063432 14657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "089ca630f03e4cf7b556735ad21d247a"
format_stamp: "Formatted at 2026-09-25 05:11:55 on dist-test-slave-1lvv"
server_key: "7aebcc4c4d917705da5a2335b0849092"
server_key_iv: "0aff41d2740301e093599d31e822db37"
server_key_version: "kuduclusterkey@0"
I20260925 05:11:55.063558 14657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:11:55.097056 14657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:11:55.099915 14657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:11:55.100162 14657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:11:55.100520 14657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:11:55.100981 14657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:11:55.101078 14657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:55.101159 14657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:11:55.101218 14657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:11:55.108896 14657 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:35621
I20260925 05:11:55.109467 14657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:11:55.110937 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 14657
I20260925 05:11:55.111042 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:11:55.111238 3098 external_mini_cluster.cc:1428] Setting key 50c1e66667bb5d2ff070091f9aaebab8
I20260925 05:11:55.112201 14785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:35621 every 8 connection(s)
Sep 25 05:11:55 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313115, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:11:55.123420 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.112071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:33299 (local address 127.3.6.190:45885)
0925 05:11:55.112118 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.112120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.112552 (+ 432us) server_negotiation.cc:407] Connection header received
0925 05:11:55.113257 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.113258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.113271 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.113302 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:55.114207 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.114980 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.116545 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.116846 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.116891 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.119826 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:55.119827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:55.119828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:55.119836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:55.121691 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:55.122221 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:55.122222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:55.122223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:55.122282 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:55.122601 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:55.122602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:55.122603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:55.122736 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:55.122793 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.123209 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.123307 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:11:55.124396 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.111778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:45885 (local address 127.3.6.131:33299)
0925 05:11:55.112424 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.112468 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:55.113130 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:55.113513 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:55.113517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:55.113786 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:11:55.114090 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.114097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.115105 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.115106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.116436 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.116439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.116993 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.116994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.117140 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.117831 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:11:55.117842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:11:55.119671 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:11:55.121793 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:55.121800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:55.121803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:55.122124 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:55.122355 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:11:55.122355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:11:55.122356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:11:55.122521 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:11:55.122829 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:11:55.122835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:11:55.123031 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 05:11:55.123958 (+ 927us) client_negotiation.cc:240] Negotiation successful
0925 05:11:55.124143 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":88,"threads_started":1}
I20260925 05:11:55.125108 14786 heartbeater.cc:347] Connected to a master server at 127.3.6.190:45885
I20260925 05:11:55.125208 14786 heartbeater.cc:464] Registering TS with master...
I20260925 05:11:55.125422 14786 heartbeater.cc:511] Master 127.3.6.190:45885 requested a full tablet report, sending...
I20260925 05:11:55.125850 14302 ts_manager.cc:194] Registered new tserver with Master: 089ca630f03e4cf7b556735ad21d247a (127.3.6.131:35621)
I20260925 05:11:55.126338 14302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:33299
I20260925 05:11:55.133584 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:11:55.151549 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.142589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.3.6.190:45885)
0925 05:11:55.142644 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.142646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.143007 (+ 361us) server_negotiation.cc:407] Connection header received
0925 05:11:55.143127 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.143128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.143140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.143200 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:11:55.144173 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.144965 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.146115 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.146399 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.146460 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.148298 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:11:55.148300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:11:55.148300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:11:55.148309 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:11:55.150116 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:55.150598 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:55.150599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:55.150599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:55.150664 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:11:55.150880 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:11:55.150881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:11:55.150881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:11:55.151014 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:11:55.151094 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.151334 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.151440 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:11:55.154053 14302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:11:55.156590 14302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:11:55.171155 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.159852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:33239 (local address 127.0.0.1:40310)
0925 05:11:55.161165 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.161208 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:55.161270 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:55.162207 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:55.162208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:55.162209 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:55.162238 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.162239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.167160 (+ 4921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.167161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.168616 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.168618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.170823 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.170824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.170916 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.170933 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 05:11:55.171009 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 05:11:55.171065 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"thread_start_us":84,"threads_started":1}
I20260925 05:11:55.172268 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.160120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40691 (local address 127.0.0.1:37596)
0925 05:11:55.161380 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.161388 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:55.161440 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:55.162057 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:55.162060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:55.162063 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:55.162133 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.162135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.164662 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.164664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.166382 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.166386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.171537 (+ 5151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.171538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.171614 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.171618 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:11:55.171670 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:11:55.171711 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":65,"threads_started":1}
I20260925 05:11:55.172829 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.160618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.3.6.129:40691)
0925 05:11:55.161834 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.161838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.161848 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:11:55.161879 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.161881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.161896 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.161973 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:55.162307 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.164552 (+ 2245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.168728 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.169595 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.169673 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.169737 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.172699 (+ 2962us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.172758 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":49,"threads_started":1}
I20260925 05:11:55.173180 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.160449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.3.6.130:33239)
0925 05:11:55.161559 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.161565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.161575 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:11:55.161615 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.161617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.161633 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.161712 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:55.163005 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.167044 (+ 4039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.169801 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.170636 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.170710 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.170785 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.173047 (+ 2262us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.173099 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":79,"threads_started":1}
I20260925 05:11:55.174072 14436 tablet_service.cc:1467] Processing CreateTablet for tablet 39a60f8311544059a43f441afeadd63c (DEFAULT_TABLE table=test-table [id=22de01a2147b4b31a70574d2e577dc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:55.174181 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 39a60f8311544059a43f441afeadd63c (DEFAULT_TABLE table=test-table [id=22de01a2147b4b31a70574d2e577dc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:55.174474 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a60f8311544059a43f441afeadd63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:55.174562 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a60f8311544059a43f441afeadd63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:55.179237 14806 tablet_bootstrap.cc:492] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: Bootstrap starting.
I20260925 05:11:55.181243 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.160308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:35621 (local address 127.0.0.1:56032)
0925 05:11:55.166529 (+ 6221us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.166541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:55.166584 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:55.168178 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:55.168179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:55.168181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:55.168243 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.168244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.169434 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.169435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.170531 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.170534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.180923 (+ 10389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.180925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.181028 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.181034 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:11:55.181101 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:11:55.181155 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6200,"thread_start_us":54,"threads_started":1}
I20260925 05:11:55.181829 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.160844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.3.6.131:35621)
0925 05:11:55.167877 (+ 7033us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.167883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.167897 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:11:55.167940 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.167942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.167960 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.168085 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:55.168320 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.169335 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.179852 (+ 10517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.180704 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.180794 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.180876 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.181676 (+ 800us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.181731 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6985,"thread_start_us":92,"threads_started":1}
I20260925 05:11:55.182035 14806 tablet_bootstrap.cc:654] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:55.182504 14806 log.cc:824] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:55.182729 14709 tablet_service.cc:1467] Processing CreateTablet for tablet 39a60f8311544059a43f441afeadd63c (DEFAULT_TABLE table=test-table [id=22de01a2147b4b31a70574d2e577dc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:11:55.183104 14709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a60f8311544059a43f441afeadd63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:11:55.183485 14806 tablet_bootstrap.cc:492] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: No bootstrap required, opened a new log
I20260925 05:11:55.183554 14806 ts_tablet_manager.cc:1397] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 05:11:55.185698 14806 raft_consensus.cc:348] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.185875 14806 raft_consensus.cc:374] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:55.185915 14806 raft_consensus.cc:729] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818a36ba6df0463f9231b2b27ba66acf, State: Initialized, Role: FOLLOWER
I20260925 05:11:55.186039 14806 consensus_queue.cc:260] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.186324 14806 ts_tablet_manager.cc:1428] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:11:55.186458 14652 heartbeater.cc:503] Master 127.3.6.190:45885 was elected leader, sending a full tablet report...
I20260925 05:11:55.182233 14805 tablet_bootstrap.cc:492] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a: Bootstrap starting.
I20260925 05:11:55.187865 14805 tablet_bootstrap.cc:654] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:55.188283 14805 log.cc:824] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:55.189248 14805 tablet_bootstrap.cc:492] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a: No bootstrap required, opened a new log
I20260925 05:11:55.189320 14805 ts_tablet_manager.cc:1397] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260925 05:11:55.189869 14809 tablet_bootstrap.cc:492] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Bootstrap starting.
I20260925 05:11:55.191416 14809 tablet_bootstrap.cc:654] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Neither blocks nor log segments found. Creating new log.
I20260925 05:11:55.191737 14809 log.cc:824] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:11:55.192086 14805 raft_consensus.cc:348] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.192265 14805 raft_consensus.cc:374] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:55.192294 14805 raft_consensus.cc:729] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109a3932c94b4645bc38d8dc4766164a, State: Initialized, Role: FOLLOWER
I20260925 05:11:55.192588 14805 consensus_queue.cc:260] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.192958 14805 ts_tablet_manager.cc:1428] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 05:11:55.193140 14514 heartbeater.cc:503] Master 127.3.6.190:45885 was elected leader, sending a full tablet report...
I20260925 05:11:55.195292 14809 tablet_bootstrap.cc:492] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: No bootstrap required, opened a new log
I20260925 05:11:55.195442 14809 ts_tablet_manager.cc:1397] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 05:11:55.197546 14809 raft_consensus.cc:348] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.197860 14809 raft_consensus.cc:374] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:11:55.197939 14809 raft_consensus.cc:729] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089ca630f03e4cf7b556735ad21d247a, State: Initialized, Role: FOLLOWER
I20260925 05:11:55.198169 14809 consensus_queue.cc:260] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.198532 14809 ts_tablet_manager.cc:1428] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:11:55.198704 14786 heartbeater.cc:503] Master 127.3.6.190:45885 was elected leader, sending a full tablet report...
Sep 25 05:11:55 dist-test-slave-1lvv krb5kdc[13826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:11:55.315682 14813 raft_consensus.cc:482] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:11:55.315887 14813 raft_consensus.cc:504] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.316361 14813 leader_election.cc:288] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818a36ba6df0463f9231b2b27ba66acf (127.3.6.130:33239), 109a3932c94b4645bc38d8dc4766164a (127.3.6.129:40691)
W20260925 05:11:55.319621 14515 tablet.cc:2369] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:55.321236 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.316644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:33239 (local address 127.3.6.131:51535)
0925 05:11:55.317123 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.317146 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:55.317196 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:55.317772 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:55.317773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:55.317775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:55.317845 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.317847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.318812 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.318813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.320006 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.320010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.320975 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.320976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.321064 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.321068 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:11:55.321119 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 05:11:55.321164 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458}
I20260925 05:11:55.321933 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.317344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:51535 (local address 127.3.6.130:33239)
0925 05:11:55.317563 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.317566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.317575 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:11:55.317604 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.317605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.317617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.317672 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:55.317939 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.318666 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.320290 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.320857 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.320930 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.320941 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.321826 (+ 885us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.321868 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260925 05:11:55.322220 14605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a60f8311544059a43f441afeadd63c" candidate_uuid: "089ca630f03e4cf7b556735ad21d247a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818a36ba6df0463f9231b2b27ba66acf" is_pre_election: true
I20260925 05:11:55.322383 14605 raft_consensus.cc:2454] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089ca630f03e4cf7b556735ad21d247a in term 0.
I20260925 05:11:55.322683 14674 leader_election.cc:302] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089ca630f03e4cf7b556735ad21d247a, 818a36ba6df0463f9231b2b27ba66acf; no voters:
I20260925 05:11:55.322955 14813 raft_consensus.cc:2781] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:11:55.323014 14813 raft_consensus.cc:482] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:11:55.323036 14813 raft_consensus.cc:3037] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:55.323992 14813 raft_consensus.cc:504] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.324131 14813 leader_election.cc:288] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [CANDIDATE]: Term 1 election: Requested vote from peers 818a36ba6df0463f9231b2b27ba66acf (127.3.6.130:33239), 109a3932c94b4645bc38d8dc4766164a (127.3.6.129:40691)
I20260925 05:11:55.324355 14605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a60f8311544059a43f441afeadd63c" candidate_uuid: "089ca630f03e4cf7b556735ad21d247a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818a36ba6df0463f9231b2b27ba66acf"
I20260925 05:11:55.324430 14605 raft_consensus.cc:3037] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:55.325352 14605 raft_consensus.cc:2454] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089ca630f03e4cf7b556735ad21d247a in term 1.
I20260925 05:11:55.325650 14674 leader_election.cc:302] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089ca630f03e4cf7b556735ad21d247a, 818a36ba6df0463f9231b2b27ba66acf; no voters:
I20260925 05:11:55.326691 14813 raft_consensus.cc:2781] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:11:55.326870 14813 raft_consensus.cc:686] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 1 LEADER]: Becoming Leader. State: Replica: 089ca630f03e4cf7b556735ad21d247a, State: Running, Role: LEADER
I20260925 05:11:55.326962 14813 consensus_queue.cc:237] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } }
I20260925 05:11:55.327972 14302 catalog_manager.cc:5495] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a reported cstate change: term changed from 0 to 1, leader changed from <none> to 089ca630f03e4cf7b556735ad21d247a (127.3.6.131). New cstate: current_term: 1 leader_uuid: "089ca630f03e4cf7b556735ad21d247a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "089ca630f03e4cf7b556735ad21d247a" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 35621 } health_report { overall_health: HEALTHY } } }
I20260925 05:11:55.331168 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.316763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40691 (local address 127.3.6.131:36177)
0925 05:11:55.321309 (+ 4546us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.321320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:11:55.321362 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:11:55.325784 (+ 4422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:11:55.325786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:11:55.325787 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:11:55.325824 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.325825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.329089 (+ 3264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.329090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.330124 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:11:55.330127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.330938 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.330939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:11:55.330996 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.330999 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:11:55.331042 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 05:11:55.331094 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4536,"thread_start_us":116,"threads_started":1}
I20260925 05:11:55.331534 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.317398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:36177 (local address 127.3.6.129:40691)
0925 05:11:55.317735 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.317737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.321417 (+ 3680us) server_negotiation.cc:407] Connection header received
0925 05:11:55.321446 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.321447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.321471 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.321515 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:11:55.325919 (+ 4404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.326588 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.330234 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.330819 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.330894 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.330906 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.331421 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.331471 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260925 05:11:55.332079 14467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a60f8311544059a43f441afeadd63c" candidate_uuid: "089ca630f03e4cf7b556735ad21d247a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109a3932c94b4645bc38d8dc4766164a" is_pre_election: true
I20260925 05:11:55.332245 14467 raft_consensus.cc:2454] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089ca630f03e4cf7b556735ad21d247a in term 0.
I20260925 05:11:55.332368 14452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a60f8311544059a43f441afeadd63c" candidate_uuid: "089ca630f03e4cf7b556735ad21d247a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109a3932c94b4645bc38d8dc4766164a"
I20260925 05:11:55.332434 14452 raft_consensus.cc:3037] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:11:55.333352 14452 raft_consensus.cc:2454] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089ca630f03e4cf7b556735ad21d247a in term 1.
W20260925 05:11:55.340363 14653 tablet.cc:2369] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:11:55.361351 14787 tablet.cc:2369] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:11:55.365556 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.361957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.3.6.131:35621)
0925 05:11:55.362148 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:11:55.362150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:11:55.362159 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:11:55.362197 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:11:55.362198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:11:55.362217 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:11:55.362284 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:11:55.362523 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.363213 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.364377 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:11:55.364693 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:11:55.364763 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:11:55.364977 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:11:55.365121 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:11:55.365206 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:11:55.365397 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 05:11:55.365435 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260925 05:11:55.370532 14605 raft_consensus.cc:1261] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 1 FOLLOWER]: Refusing update from remote peer 089ca630f03e4cf7b556735ad21d247a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:55.370532 14452 raft_consensus.cc:1261] T 39a60f8311544059a43f441afeadd63c P 109a3932c94b4645bc38d8dc4766164a [term 1 FOLLOWER]: Refusing update from remote peer 089ca630f03e4cf7b556735ad21d247a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:11:55.370924 14815 consensus_queue.cc:1035] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818a36ba6df0463f9231b2b27ba66acf" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 33239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:55.371210 14815 consensus_queue.cc:1035] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [LEADER]: Connected to new peer: Peer: permanent_uuid: "109a3932c94b4645bc38d8dc4766164a" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:11:55.374040 14818 mvcc.cc:205] Tried to move back new op lower bound from 7333122520554397696 to 7333122520379940864. Current Snapshot: MvccSnapshot[applied={T|T < 7333122520554397696}]
I20260925 05:11:55.374922 14820 mvcc.cc:205] Tried to move back new op lower bound from 7333122520554397696 to 7333122520379940864. Current Snapshot: MvccSnapshot[applied={T|T < 7333122520554397696}]
I20260925 05:11:55.375177 14819 mvcc.cc:205] Tried to move back new op lower bound from 7333122520554397696 to 7333122520379940864. Current Snapshot: MvccSnapshot[applied={T|T < 7333122520554397696}]
I20260925 05:11:55.377715 14301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:55.377816 14301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:11:55.378515 14301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52ff43ab219546a1a35bcd5ff989fac1: Sending DeleteTablet for 3 replicas of tablet 39a60f8311544059a43f441afeadd63c
I20260925 05:11:55.379045 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a60f8311544059a43f441afeadd63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:55 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:40310
I20260925 05:11:55.379240 14436 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a60f8311544059a43f441afeadd63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:55 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:37596
I20260925 05:11:55.379447 14709 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a60f8311544059a43f441afeadd63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:11:55 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:56032
I20260925 05:11:55.380106 14827 tablet_replica.cc:330] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: stopping tablet replica
I20260925 05:11:55.380200 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 14383
I20260925 05:11:55.380992 14829 tablet_replica.cc:330] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: stopping tablet replica
I20260925 05:11:55.381184 14827 raft_consensus.cc:2229] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:11:55.381342 14829 raft_consensus.cc:2229] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:11:55.381560 14829 raft_consensus.cc:2258] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:55.381750 14827 raft_consensus.cc:2258] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:11:55.382402 14829 ts_tablet_manager.cc:1905] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:55.384598 14829 ts_tablet_manager.cc:1918] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:11:55.384732 14829 log.cc:1196] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/39a60f8311544059a43f441afeadd63c
I20260925 05:11:55.385109 14829 ts_tablet_manager.cc:1939] T 39a60f8311544059a43f441afeadd63c P 089ca630f03e4cf7b556735ad21d247a: Deleting consensus metadata
I20260925 05:11:55.385622 14288 catalog_manager.cc:4841] TS 089ca630f03e4cf7b556735ad21d247a (127.3.6.131:35621): tablet 39a60f8311544059a43f441afeadd63c (table test-table [id=22de01a2147b4b31a70574d2e577dc8a]) successfully deleted
I20260925 05:11:55.386276 14827 ts_tablet_manager.cc:1905] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:11:55.388028 14827 ts_tablet_manager.cc:1918] T 39a60f8311544059a43f441afeadd63c P 818a36ba6df0463f9231b2b27ba66acf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 05:11:55.388262 14286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:40691 (error 108)
I20260925 05:11:55.389462 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.389313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40691 (local address 127.0.0.1:37604)
0925 05:11:55.389382 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.389428 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 05:11:55.389704 14286 catalog_manager.cc:4584] TS 109a3932c94b4645bc38d8dc4766164a (127.3.6.129:40691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39a60f8311544059a43f441afeadd63c: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40691: connect: Connection refused (error 111)
I20260925 05:11:55.389933 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 14520
I20260925 05:11:55.396929 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:11:55.396817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:33239 (local address 127.0.0.1:40312)
0925 05:11:55.396865 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 05:11:55.396888 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.130:33239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 05:11:55.397250 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 14657
I20260925 05:11:55.403678 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 14270
I20260925 05:11:55.411037 3098 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:11:55.756067 3098 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:11:55.756179 3098 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:11:56.101094 3098 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:11:56.101 UTC [13856] LOG: received smart shutdown request
2026-09-25 05:11:56.105 UTC [13856] LOG: background worker "logical replication launcher" (PID 13865) exited with exit code 1
2026-09-25 05:11:56.105 UTC [13860] LOG: shutting down
2026-09-25 05:11:56.296 UTC [13856] LOG: database system is shut down
2026-09-25T05:11:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48515 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:11:56 dist-test-slave-1lvv krb5kdc[14839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:11:56 dist-test-slave-1lvv krb5kdc[14839](info): set up 2 sockets
Sep 25 05:11:56 dist-test-slave-1lvv krb5kdc[14839](info): commencing operation
krb5kdc: starting...
W20260925 05:11:57.467826 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:11:57 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:11:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/rangeradmin_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/rangeradmin_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/rangerlookup_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/rangerlookup_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
I20260925 05:11:57.648912 3098 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskH05q1Z/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/postgres -l logfile start
2026-09-25 05:12:00.373 UTC [14869] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:12:00.373 UTC [14869] LOG: listening on IPv4 address "127.3.6.148", port 41121
2026-09-25 05:12:00.377 UTC [14869] LOG: listening on Unix socket "/tmp/.s.PGSQL.41121"
2026-09-25 05:12:00.400 UTC [14872] LOG: database system was shut down at 2026-09-25 05:11:58 UTC
2026-09-25 05:12:00.404 UTC [14869] LOG: database system is ready to accept connections
I20260925 05:12:01.379810 3098 mini_postgres.cc:96] Postgres bound to 41121
2026-09-25 05:12:01.386 UTC [14880] FATAL: database "slave" does not exist
127.3.6.148:41121 - accepting connections
I20260925 05:12:01.387202 3098 mini_ranger.cc:161] Starting Ranger...
I20260925 05:12:01.403162 3098 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:12:01.949033 3098 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:12:01.949110 3098 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-admin
2026-09-25 05:12:02,238 [I] DB FLAVOR :POSTGRES
2026-09-25 05:12:02,239 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:12:02,239 [I] Checking connection..
2026-09-25 05:12:02,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:12:02,491 [I] Checking connection passed.
2026-09-25 05:12:02,492 [I] --------- Verifying version history table ---------
2026-09-25 05:12:02,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:12:02,758 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:12:02,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:12:03,001 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:12:03,001 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:12:03,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:12:03.266 UTC [14944] WARNING: there is no transaction in progress
2026-09-25 05:12:03,272 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:12:03,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:12:03,528 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:12:03,528 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:12:03,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:12:03,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:12:04,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1lvv.c.gcp-upstream.internal','N') ;"
2026-09-25 05:12:04,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:12:04,541 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:12:04,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:12:04,802 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:12:04,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:12:05,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:12:05,293 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:12:05,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:12:06.304 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.321 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.339 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.357 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.375 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.397 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.460 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.468 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.484 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.496 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.509 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.524 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.535 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.542 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.551 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:06.559 UTC [15052] WARNING: there is no transaction in progress
2026-09-25 05:12:07,265 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:12:07,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:12:07,524 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:12:07,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:12:07,795 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:12:07,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:12:08,030 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:12:08,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:12:08,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:12:08,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1lvv.c.gcp-upstream.internal';"
2026-09-25 05:12:08,765 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:12:08,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:12:09,028 [I] DB_PATCHES have already been applied
I20260925 05:12:09.034721 3098 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-admin:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskH05q1Z/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:12:09.034821 3098 mini_ranger.cc:193] Using host: 127.3.6.148
I20260925 05:12:09.038955 3098 mini_ranger.cc:227] Ranger admin URL: http://127.3.6.148:42789
Sep 25, 2026 5:12:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:12:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:12:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:12:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:12:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:12:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:12:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:12:10.129393 3098 mini_ranger.cc:160] Time spent starting Ranger: real 8.742s user 0.000s sys 0.010s
Sep 25, 2026 5:12:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:12:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 5:12:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:12:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:12:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:12:31.788754 3098 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.6.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.6.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/rangerkms_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.6.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/rangerkms_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.6.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.6.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.6.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/HTTP_127.3.6.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:12:31 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313151, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:12:31.888164 3098 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:12:31.898927 3098 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:12:32.872285 3098 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:12:32.872407 3098 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms
I20260925 05:12:32.872428 3098 mini_ranger_kms.cc:227] Using postgres at 127.3.6.148:41121
2026-09-25 05:12:33,057 [I] DB FLAVOR :POSTGRES
2026-09-25 05:12:33,057 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:12:33,057 [I] Checking connection
2026-09-25 05:12:33,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:12:33,304 [I] connection success
2026-09-25 05:12:33,304 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:12:33,304 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:12:33,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:12:33,544 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:12:33,544 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:12:33,545 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:12:33,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.6.148:41121/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:12:33,834 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:12:34.322669 3098 mini_ranger_kms.cc:310] Created kms service
I20260925 05:12:34.526669 3098 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:12:34.575786 3098 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:12:34 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313151, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:34.976902 3098 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:12:34.977147 3098 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms:/tmp/dist-test-taskH05q1Z/build/release/bin/postgresql.jar:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskH05q1Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:12:34.977173 3098 mini_ranger_kms.cc:240] Using host: 127.3.6.148
I20260925 05:12:34.981763 3098 mini_ranger_kms.cc:276] Ranger KMS PID: 15232
I20260925 05:12:34.981874 3098 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.6.148:57163
Sep 25, 2026 5:12:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:12:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:12:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:12:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:12:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:12:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 05:12:37.291719 3098 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.404s user 0.007s sys 0.161s
I20260925 05:12:37.291893 3098 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:12:37.291963 3098 mini_ranger_kms.cc:401] 127.3.6.148:57163/kms/v1/keys
Sep 25 05:12:37 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790313157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.6.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 05:12:37 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.6.148@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
Sep 25 05:12:38 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.6.148@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
Sep 25 05:12:40 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:12:40.723093 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44377
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44377
--ranger_config_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:40.818794 15283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:40.819033 15283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:40.819056 15283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:40.820196 15283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:40.820240 15283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:40.820254 15283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:40.820268 15283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:40.820281 15283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:40.821867 15283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.6.148:57163/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44377
--ranger_config_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:40.822079 15283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:40.822268 15283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:40.827734 15289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:40.827792 15291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:40.827934 15283 server_base.cc:1034] running on GCE node
W20260925 05:12:40.827930 15288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:40.828403 15283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:40.828727 15283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:40.829887 15283 hybrid_clock.cc:648] HybridClock initialized: now 1790313160829877 us; error 36 us; skew 500 ppm
Sep 25 05:12:40 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:40.840094 15283 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:12:40.840644 15283 webserver.cc:460] Webserver started at http://127.3.6.190:46501/ using document root <none> and password file <none>
I20260925 05:12:40.840852 15283 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:40.840891 15283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:40.840992 15283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:12:40 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:40.911581 15283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "e53e2426376142b89c544aacb4f3f4d9"
format_stamp: "Formatted at 2026-09-25 05:12:40 on dist-test-slave-1lvv"
server_key: "e1915497ad8d9915aba872d51acc5d23"
server_key_iv: "881d723dceae7af2cd1842d8d8479c78"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:40.912209 15283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "e53e2426376142b89c544aacb4f3f4d9"
format_stamp: "Formatted at 2026-09-25 05:12:40 on dist-test-slave-1lvv"
server_key: "e1915497ad8d9915aba872d51acc5d23"
server_key_iv: "881d723dceae7af2cd1842d8d8479c78"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:40.914362 15283 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:12:40.930550 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:40.930846 15283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:40.930977 15283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "e53e2426376142b89c544aacb4f3f4d9"
format_stamp: "Formatted at 2026-09-25 05:12:40 on dist-test-slave-1lvv"
server_key: "e1915497ad8d9915aba872d51acc5d23"
server_key_iv: "881d723dceae7af2cd1842d8d8479c78"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:40.931097 15283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:40.946539 15283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:40.948720 15283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:40.948870 15283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:40.953382 15283 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44377
I20260925 05:12:40.953428 15350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44377 every 8 connection(s)
I20260925 05:12:40.953805 15283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:40.954442 15351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:40.956925 15351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Bootstrap starting.
I20260925 05:12:40.957540 15351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:40.957851 15351 log.cc:824] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:40.958546 15351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: No bootstrap required, opened a new log
I20260925 05:12:40.961068 15351 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:40.961253 15351 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:40.961282 15351 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e2426376142b89c544aacb4f3f4d9, State: Initialized, Role: FOLLOWER
I20260925 05:12:40.961472 15351 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:40.961567 15351 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:40.961617 15351 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:40.961675 15351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:40.962404 15351 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:40.962546 15351 leader_election.cc:302] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53e2426376142b89c544aacb4f3f4d9; no voters:
I20260925 05:12:40.962604 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 15283
I20260925 05:12:40.962716 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:12:40.962779 15351 leader_election.cc:288] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:40.962875 15354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:40.962970 3098 external_mini_cluster.cc:1428] Setting key cbbb7ebd87a7b33f818258ff30e67709
I20260925 05:12:40.963109 15354 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 LEADER]: Becoming Leader. State: Replica: e53e2426376142b89c544aacb4f3f4d9, State: Running, Role: LEADER
I20260925 05:12:40.963150 15351 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:40.963198 15354 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:40.963552 15354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53e2426376142b89c544aacb4f3f4d9. Latest consensus state: current_term: 1 leader_uuid: "e53e2426376142b89c544aacb4f3f4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } } }
I20260925 05:12:40.963629 15354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:40.963702 15356 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53e2426376142b89c544aacb4f3f4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } } }
I20260925 05:12:40.963836 15356 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:40.963886 15360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:40.964502 15360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:40.965054 15351 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:12:40.970755 15360 catalog_manager.cc:1347] Generated new cluster ID: 2093761a62d6466f9610e7cb5f2dcc58
I20260925 05:12:40.970851 15360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:12:40 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:40.980072 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:40.963551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.3.6.190:44377)
0925 05:12:40.964210 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:40.964224 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:40.964265 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:12:40.964402 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:40.964411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:40.964480 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:40.964654 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:40.965736 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:40.970249 (+ 4513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:40.971457 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:40.972138 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:40.975433 (+ 3295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:40.975554 (+ 121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:40.975558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:40.975558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:40.975568 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:40.977773 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:40.978239 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:40.978242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:40.978243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:40.978362 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:40.978565 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:40.978566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:40.978566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:40.978911 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:40.979532 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:40.979612 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:12:40.979791 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":72,"threads_started":1}
I20260925 05:12:40.985781 15360 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:40.986449 15360 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:40.989235 15360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Generated new TSK 0
I20260925 05:12:40.989449 15360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:12:41.678465 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44377
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:41.793671 15396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:41.794015 15396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:41.794055 15396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:41.795990 15396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:41.796073 15396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:41.796120 15396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:12:41.797986 15396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.6.148:57163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:41.798332 15396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:41.798604 15396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:41.805374 15401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:41.805526 15404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:41.805884 15402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:41.806303 15396 server_base.cc:1034] running on GCE node
I20260925 05:12:41.806545 15396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:41.806887 15396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:41.808073 15396 hybrid_clock.cc:648] HybridClock initialized: now 1790313161808060 us; error 33 us; skew 500 ppm
Sep 25 05:12:41 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:41.811646 15396 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:12:41.812292 15396 webserver.cc:460] Webserver started at http://127.3.6.129:40781/ using document root <none> and password file <none>
I20260925 05:12:41.812615 15396 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:41.812745 15396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:41.812922 15396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:12:41 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:41.829284 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "d030b2676d684af3a0707593a8005782"
format_stamp: "Formatted at 2026-09-25 05:12:41 on dist-test-slave-1lvv"
server_key: "d03def1b60c065feabab93d997256f39"
server_key_iv: "fc695db8667bc97de44bb2960b7609d2"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:41.829974 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "d030b2676d684af3a0707593a8005782"
format_stamp: "Formatted at 2026-09-25 05:12:41 on dist-test-slave-1lvv"
server_key: "d03def1b60c065feabab93d997256f39"
server_key_iv: "fc695db8667bc97de44bb2960b7609d2"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:41.832453 15396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 05:12:41.848623 15413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:41.849001 15396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:12:41.849383 15396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "d030b2676d684af3a0707593a8005782"
format_stamp: "Formatted at 2026-09-25 05:12:41 on dist-test-slave-1lvv"
server_key: "d03def1b60c065feabab93d997256f39"
server_key_iv: "fc695db8667bc97de44bb2960b7609d2"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:41.849660 15396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:41.869948 15396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:41.872339 15396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:41.872514 15396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:41.872792 15396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:41.873215 15396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:41.873270 15396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:41.873339 15396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:41.873386 15396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:41.881261 15396 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:45095
I20260925 05:12:41.881814 15396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:12:41.884097 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 15396
I20260925 05:12:41.884205 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:12:41.884451 3098 external_mini_cluster.cc:1428] Setting key fa17c5314aea4fd48181b9f3bd0f4513
I20260925 05:12:41.889719 15526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:45095 every 8 connection(s)
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Sep 25 05:12:41 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:41.918131 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:41.890647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44377 (local address 127.3.6.129:38033)
0925 05:12:41.891197 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:41.891241 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:41.891896 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:41.892335 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:41.892339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:41.892604 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:41.892896 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:41.892902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:41.893932 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:41.893933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:41.895233 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:41.895236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:41.895931 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:41.895932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:41.896094 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:41.908213 (+ 12119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:41.908230 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:41.913704 (+ 5474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:41.916175 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:41.916189 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:41.916192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:41.916582 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:41.916843 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:41.916844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:41.916845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:41.917011 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:41.917347 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:41.917357 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:41.917573 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 05:12:41.917694 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 05:12:41.917875 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":123,"threads_started":1}
I20260925 05:12:41.918978 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:41.890999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:38033 (local address 127.3.6.190:44377)
0925 05:12:41.892087 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:41.892090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:41.892098 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:12:41.892124 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:41.892125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:41.892137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:41.892208 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:41.893008 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:41.893824 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:41.895362 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:41.895699 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:41.895832 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:41.913911 (+ 18079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:41.913914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:41.913915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:41.913925 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:41.916047 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:41.916692 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:41.916693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:41.916694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:41.916767 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:41.917083 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:41.917084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:41.917084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:41.917239 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:41.917315 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:41.918768 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0925 05:12:41.918868 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":83,"threads_started":1}
I20260925 05:12:41.919478 15527 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44377
I20260925 05:12:41.919580 15527 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:41.919811 15527 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:41.920404 15315 ts_manager.cc:194] Registered new tserver with Master: d030b2676d684af3a0707593a8005782 (127.3.6.129:45095)
I20260925 05:12:41.921312 15315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:38033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:12:41.938989 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44377
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:12:41 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790313161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 05:12:42.070993 15535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:42.071293 15535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:42.071334 15535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:42.073161 15535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:42.073261 15535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:42.073323 15535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:12:42.075554 15535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.6.148:57163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:42.075848 15535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:42.076110 15535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:42.082792 15544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.082932 15542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.082875 15541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:42.083590 15535 server_base.cc:1034] running on GCE node
I20260925 05:12:42.084048 15535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:42.084323 15535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:42.085482 15535 hybrid_clock.cc:648] HybridClock initialized: now 1790313162085461 us; error 36 us; skew 500 ppm
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:42.088984 15535 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:12:42.089551 15535 webserver.cc:460] Webserver started at http://127.3.6.130:38751/ using document root <none> and password file <none>
I20260925 05:12:42.089854 15535 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:42.089941 15535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:42.090108 15535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:42.103828 15535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "f0ec450ae9604d5a8a68bf81df950bcb"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "03ef71c9ba2e5886b5c3e0e607cd3b08"
server_key_iv: "a3dc8d607408b94e398e92c2661ef968"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.104312 15535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "f0ec450ae9604d5a8a68bf81df950bcb"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "03ef71c9ba2e5886b5c3e0e607cd3b08"
server_key_iv: "a3dc8d607408b94e398e92c2661ef968"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.108266 15535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260925 05:12:42.123940 15551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.124245 15535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 05:12:42.124368 15535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "f0ec450ae9604d5a8a68bf81df950bcb"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "03ef71c9ba2e5886b5c3e0e607cd3b08"
server_key_iv: "a3dc8d607408b94e398e92c2661ef968"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.124473 15535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:42.150859 15535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:42.153743 15535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:42.153990 15535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:42.154332 15535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:42.154865 15535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:42.154963 15535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.155043 15535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:42.155107 15535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.162042 15535 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:35757
I20260925 05:12:42.162765 15535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:12:42.166669 15664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:35757 every 8 connection(s)
I20260925 05:12:42.166777 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 15535
I20260925 05:12:42.166884 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:12:42.167104 3098 external_mini_cluster.cc:1428] Setting key 29c55be3900472ac9fe9cacc2de71122
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:42.183503 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:42.167831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:60279 (local address 127.3.6.190:44377)
0925 05:12:42.167879 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:42.167881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:42.169856 (+ 1975us) server_negotiation.cc:407] Connection header received
0925 05:12:42.170547 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:42.170549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:42.170561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:42.170592 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:42.171441 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.172258 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.173794 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.174122 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.174167 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:42.179717 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:42.179719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:42.179720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:42.179729 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:42.181695 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:42.182276 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:42.182278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:42.182278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:42.182343 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:42.182637 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:42.182638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:42.182639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:42.182798 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:42.182846 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:42.183279 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 05:12:42.183386 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:12:42.184365 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:42.167495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44377 (local address 127.3.6.130:60279)
0925 05:12:42.167700 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:42.167746 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:42.170422 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:42.170792 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:42.170797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:42.171047 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:42.171327 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:42.171333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.172387 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.172388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:42.173675 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:42.173678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.174271 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.174272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:42.174426 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:42.177710 (+ 3284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:42.177722 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:42.179565 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:42.181825 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:42.181835 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:42.181838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:42.182174 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:42.182417 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:42.182418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:42.182418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:42.182563 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:42.182899 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:42.182906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:42.183109 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 05:12:42.183935 (+ 826us) client_negotiation.cc:240] Negotiation successful
0925 05:12:42.184119 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":125,"threads_started":1}
I20260925 05:12:42.185091 15665 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44377
I20260925 05:12:42.185189 15665 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:42.185384 15665 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:42.185732 15315 ts_manager.cc:194] Registered new tserver with Master: f0ec450ae9604d5a8a68bf81df950bcb (127.3.6.130:35757)
I20260925 05:12:42.186233 15315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:60279
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:12:42.215962 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44377
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:42.319269 15670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:42.319525 15670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:42.319567 15670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:42.320884 15670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:42.320964 15670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:42.321012 15670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:12:42.322727 15670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.6.148:57163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:42.323110 15670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:42.323396 15670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:42.330623 15676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:42.330964 15670 server_base.cc:1034] running on GCE node
W20260925 05:12:42.331326 15678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.332408 15675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:42.332757 15670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:42.333079 15670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:42.340443 15670 hybrid_clock.cc:648] HybridClock initialized: now 1790313162340420 us; error 46 us; skew 500 ppm
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:42.344071 15670 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:12:42.344628 15670 webserver.cc:460] Webserver started at http://127.3.6.131:32815/ using document root <none> and password file <none>
I20260925 05:12:42.344940 15670 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:42.344993 15670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:42.345119 15670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:42.371001 15670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "752b6698f1594a9c92a65dd440644d54"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "7f803f076dbea772fc5db3d7623d9be3"
server_key_iv: "117c7cf6e9ba76fd8b11b6a582abf64e"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.371630 15670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "752b6698f1594a9c92a65dd440644d54"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "7f803f076dbea772fc5db3d7623d9be3"
server_key_iv: "117c7cf6e9ba76fd8b11b6a582abf64e"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.378762 15670 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 05:12:42.398538 15685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.398909 15670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 05:12:42.399041 15670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "752b6698f1594a9c92a65dd440644d54"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "7f803f076dbea772fc5db3d7623d9be3"
server_key_iv: "117c7cf6e9ba76fd8b11b6a582abf64e"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.399170 15670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:42.427723 15670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:42.430358 15670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:42.430572 15670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:42.430889 15670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:42.431274 15670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:42.431361 15670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.431432 15670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:42.431488 15670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.438428 15670 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:46269
I20260925 05:12:42.439044 15670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:12:42.441859 15798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:46269 every 8 connection(s)
I20260925 05:12:42.443962 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 15670
I20260925 05:12:42.444062 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:12:42.444258 3098 external_mini_cluster.cc:1428] Setting key 55aa152d47948d58d67799fd4817b1c9
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:42.453931 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:42.442736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:38395 (local address 127.3.6.190:44377)
0925 05:12:42.442796 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:42.442798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:42.443293 (+ 495us) server_negotiation.cc:407] Connection header received
0925 05:12:42.444078 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:42.444080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:42.444093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:42.444129 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:42.445493 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.446254 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.447726 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.448049 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.448092 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:42.450691 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:42.450692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:42.450693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:42.450700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:42.452411 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:42.452888 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:42.452889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:42.452890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:42.452946 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:42.453200 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:42.453201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:42.453201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:42.453322 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:42.453361 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:42.453733 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 05:12:42.453826 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:12:42.454833 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:42.442446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44377 (local address 127.3.6.131:38395)
0925 05:12:42.443158 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:42.443201 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:42.443932 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:42.444340 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:42.444345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:42.445062 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:42.445376 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:42.445383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.446374 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.446375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:42.447624 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:42.447626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.448200 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.448201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:42.448343 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:42.448986 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:42.448997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:42.450585 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:42.452506 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:42.452513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:42.452515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:42.452802 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:42.453012 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:42.453013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:42.453013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:42.453133 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:42.453406 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:42.453411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:42.453586 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 05:12:42.454339 (+ 753us) client_negotiation.cc:240] Negotiation successful
0925 05:12:42.454575 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":80,"threads_started":1}
I20260925 05:12:42.455611 15799 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44377
I20260925 05:12:42.455710 15799 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:42.455960 15799 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:42.456380 15314 ts_manager.cc:194] Registered new tserver with Master: 752b6698f1594a9c92a65dd440644d54 (127.3.6.131:46269)
I20260925 05:12:42.456772 15314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:38395
I20260925 05:12:42.466813 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:12:42.467748 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 15396
I20260925 05:12:42.472723 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 15535
I20260925 05:12:42.480316 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 15670
I20260925 05:12:42.486425 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 15283
I20260925 05:12:42.493093 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44377
--webserver_interface=127.3.6.190
--webserver_port=46501
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44377
--ranger_config_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:42.578945 15802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:42.579178 15802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:42.579201 15802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:42.580494 15802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:42.580546 15802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:42.580561 15802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:42.580574 15802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:42.580587 15802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:42.582265 15802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44377
--ranger_config_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:42.582528 15802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:42.582778 15802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:42.587741 15810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.587808 15807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.587746 15808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:42.588128 15802 server_base.cc:1034] running on GCE node
I20260925 05:12:42.588373 15802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:42.588680 15802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:42.589824 15802 hybrid_clock.cc:648] HybridClock initialized: now 1790313162589814 us; error 38 us; skew 500 ppm
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:42.592526 15802 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:12:42.593022 15802 webserver.cc:460] Webserver started at http://127.3.6.190:46501/ using document root <none> and password file <none>
I20260925 05:12:42.593276 15802 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:42.593345 15802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:42.653028 15802 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:42.654008 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.654279 15802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:42.654388 15802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "e53e2426376142b89c544aacb4f3f4d9"
format_stamp: "Formatted at 2026-09-25 05:12:40 on dist-test-slave-1lvv"
server_key: "e1915497ad8d9915aba872d51acc5d23"
server_key_iv: "881d723dceae7af2cd1842d8d8479c78"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.654867 15802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:42.670431 15802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:42.672638 15802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:42.672819 15802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:42.677981 15802 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44377
I20260925 05:12:42.678017 15870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44377 every 8 connection(s)
I20260925 05:12:42.678501 15802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:42.679847 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 15802
I20260925 05:12:42.680056 15871 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:12:42.680174 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:45095
--local_ip_for_outbound_sockets=127.3.6.129
--tserver_master_addrs=127.3.6.190:44377
--webserver_port=40781
--webserver_interface=127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:42.681160 15871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Bootstrap starting.
I20260925 05:12:42.689698 15871 log.cc:824] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:42.691581 15871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:12:42.691938 15871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Bootstrap complete.
I20260925 05:12:42.694373 15871 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:42.694621 15871 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e2426376142b89c544aacb4f3f4d9, State: Initialized, Role: FOLLOWER
I20260925 05:12:42.694831 15871 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:42.694933 15871 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:42.694999 15871 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:42.695051 15871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:12:42.696329 15871 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:42.696463 15871 leader_election.cc:302] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53e2426376142b89c544aacb4f3f4d9; no voters:
I20260925 05:12:42.696681 15871 leader_election.cc:288] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:12:42.696723 15875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:12:42.696889 15875 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [term 2 LEADER]: Becoming Leader. State: Replica: e53e2426376142b89c544aacb4f3f4d9, State: Running, Role: LEADER
I20260925 05:12:42.697003 15871 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:42.697095 15875 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } }
I20260925 05:12:42.697507 15875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53e2426376142b89c544aacb4f3f4d9. Latest consensus state: current_term: 2 leader_uuid: "e53e2426376142b89c544aacb4f3f4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } } }
I20260925 05:12:42.697600 15877 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e53e2426376142b89c544aacb4f3f4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e2426376142b89c544aacb4f3f4d9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44377 } } }
I20260925 05:12:42.697711 15875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:42.697721 15877 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:42.697953 15884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:42.698630 15884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:42.698834 15884 catalog_manager.cc:1259] Loaded cluster ID: 2093761a62d6466f9610e7cb5f2dcc58
I20260925 05:12:42.698917 15884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:42.699160 15871 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:12:42.702919 15884 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:42.703531 15884 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Loaded TSK: 0
I20260925 05:12:42.703943 15884 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:12:42.782864 15873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:42.783114 15873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:42.783159 15873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:42.784759 15873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:42.784840 15873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:42.784890 15873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:12:42.787123 15873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:45095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=40781
--tserver_master_addrs=127.3.6.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:42.787415 15873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:42.787683 15873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:42.793305 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.793207 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.793447 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:42.793926 15873 server_base.cc:1034] running on GCE node
I20260925 05:12:42.796914 15873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:42.797290 15873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:42.798489 15873 hybrid_clock.cc:648] HybridClock initialized: now 1790313162798457 us; error 68 us; skew 500 ppm
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:42.801692 15873 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:12:42.802297 15873 webserver.cc:460] Webserver started at http://127.3.6.129:40781/ using document root <none> and password file <none>
I20260925 05:12:42.802588 15873 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:42.802678 15873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:42.836937 15873 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:42.838706 15919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.839025 15873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:12:42.839143 15873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "d030b2676d684af3a0707593a8005782"
format_stamp: "Formatted at 2026-09-25 05:12:41 on dist-test-slave-1lvv"
server_key: "d03def1b60c065feabab93d997256f39"
server_key_iv: "fc695db8667bc97de44bb2960b7609d2"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:42.839648 15873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:42.855801 15873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:42.858409 15873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:42.858568 15873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:42.858834 15873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:42.859229 15873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:42.859328 15873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.859436 15873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:42.859513 15873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:42.866487 15873 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:45095
I20260925 05:12:42.867102 15873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:12:42.867614 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 15873
I20260925 05:12:42.867939 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:35757
--local_ip_for_outbound_sockets=127.3.6.130
--tserver_master_addrs=127.3.6.190:44377
--webserver_port=38751
--webserver_interface=127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:42.869202 16032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:45095 every 8 connection(s)
Sep 25 05:12:42 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:42.887992 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:42.872833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:50469 (local address 127.3.6.190:44377)
0925 05:12:42.873929 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:42.873938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:42.873974 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:12:42.874195 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:42.874201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:42.874251 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:42.874380 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:42.876952 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.878356 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.880045 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.880706 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.880844 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:42.883509 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:42.883515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:42.883515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:42.883529 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:42.885685 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:42.886287 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:42.886291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:42.886291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:42.886417 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:42.886757 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:42.886757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:42.886758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:42.887109 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:42.887196 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:42.887616 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 05:12:42.887777 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":91,"threads_started":1}
I20260925 05:12:42.887995 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:42.872755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44377 (local address 127.3.6.129:50469)
0925 05:12:42.873041 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:42.873091 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:42.873717 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:42.874528 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:42.874532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:42.874766 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:42.875020 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:42.875024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.878496 (+ 3472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.878497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:42.879917 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:42.879921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:42.880969 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:42.880971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:42.881112 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:42.881754 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:42.881777 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:42.883365 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:42.885793 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:42.885802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:42.885812 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:42.886191 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:42.886507 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:42.886508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:42.886509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:42.886674 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:42.887228 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:42.887235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:42.887398 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 05:12:42.887525 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 05:12:42.887704 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
I20260925 05:12:42.888942 16033 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44377
I20260925 05:12:42.889047 16033 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:42.889249 16033 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
W20260925 05:12:42.890239 16033 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:44377 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:12:42.890321 16033 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:42.890391 16033 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:42.890720 16033 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:42.890815 16033 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
W20260925 05:12:42.891142 16033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:12:42.988345 16034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:42.988749 16034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:42.988811 16034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:42.990495 16034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:42.990576 16034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:42.990629 16034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:12:42.992550 16034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:35757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=38751
--tserver_master_addrs=127.3.6.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:42.992846 16034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:42.993140 16034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:42.999185 16042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:42.999390 16045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:43.001384 16034 server_base.cc:1034] running on GCE node
W20260925 05:12:43.007946 16043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:43.008376 16034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:43.008793 16034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:43.010008 16034 hybrid_clock.cc:648] HybridClock initialized: now 1790313163009991 us; error 34 us; skew 500 ppm
Sep 25 05:12:43 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:43.013622 16034 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:12:43.014251 16034 webserver.cc:460] Webserver started at http://127.3.6.130:38751/ using document root <none> and password file <none>
I20260925 05:12:43.014549 16034 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:43.014652 16034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:12:43 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:43.035979 16034 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 05:12:43.036978 16053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:43.037195 16034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:12:43.037287 16034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "f0ec450ae9604d5a8a68bf81df950bcb"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "03ef71c9ba2e5886b5c3e0e607cd3b08"
server_key_iv: "a3dc8d607408b94e398e92c2661ef968"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:43.038507 16034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:43.055586 16034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:43.059088 16034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:43.059293 16034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:43.059590 16034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:43.060032 16034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:43.060087 16034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:43.060120 16034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:43.060145 16034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:43.067374 16034 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:35757
I20260925 05:12:43.067526 16166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:35757 every 8 connection(s)
I20260925 05:12:43.067951 16034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:12:43.069797 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16034
I20260925 05:12:43.070108 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.6.148:57163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:46269
--local_ip_for_outbound_sockets=127.3.6.131
--tserver_master_addrs=127.3.6.190:44377
--webserver_port=32815
--webserver_interface=127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:12:43 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:43.080822 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.069164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:49095 (local address 127.3.6.190:44377)
0925 05:12:43.069251 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:43.069253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:43.069265 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:12:43.069770 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:43.069772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:43.069792 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:43.069827 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:43.070739 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.071508 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.073017 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.073317 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.073403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.076267 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:43.076269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:43.076270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:43.076279 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:43.078718 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.079394 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.079396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.079396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.079472 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.079878 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.079879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.079880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.080056 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:43.080142 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:43.080602 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 05:12:43.080719 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:12:43.081516 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.068852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44377 (local address 127.3.6.130:49095)
0925 05:12:43.069021 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.069062 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:43.069626 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:43.070042 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:43.070046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:43.070320 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:43.070614 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.070621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.071632 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.071633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.072914 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.072916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.073449 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.073451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.073603 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.074267 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:43.074278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:43.076138 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:43.078858 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.078869 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.078872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.079279 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.079560 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.079561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.079561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.079742 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.080177 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:43.080186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:43.080405 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 05:12:43.080995 (+ 590us) client_negotiation.cc:240] Negotiation successful
0925 05:12:43.081201 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260925 05:12:43.084501 16167 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44377
I20260925 05:12:43.084633 16167 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:43.084882 16167 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
W20260925 05:12:43.085701 16167 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:44377 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:12:43.085793 16167 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:43.085875 16167 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:43.086270 16167 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:43.086359 16167 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
W20260925 05:12:43.086719 16167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:12:43.175616 16170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:43.175904 16170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:43.175946 16170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:43.177356 16170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:43.177423 16170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:43.177466 16170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:12:43.179461 16170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.6.148:57163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:46269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=32815
--tserver_master_addrs=127.3.6.190:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:43.179759 16170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:43.180027 16170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:43.185015 16175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:43.185465 16178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:43.185469 16170 server_base.cc:1034] running on GCE node
W20260925 05:12:43.188292 16176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:43.188539 16170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:43.188833 16170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:43.196187 16170 hybrid_clock.cc:648] HybridClock initialized: now 1790313163196131 us; error 63 us; skew 500 ppm
Sep 25 05:12:43 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:43.199427 16170 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:12:43.200048 16170 webserver.cc:460] Webserver started at http://127.3.6.131:32815/ using document root <none> and password file <none>
I20260925 05:12:43.200297 16170 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:43.200388 16170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:12:43 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:43.218086 16170 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 05:12:43.219358 16187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:43.219588 16170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:12:43.219835 16170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "752b6698f1594a9c92a65dd440644d54"
format_stamp: "Formatted at 2026-09-25 05:12:42 on dist-test-slave-1lvv"
server_key: "7f803f076dbea772fc5db3d7623d9be3"
server_key_iv: "117c7cf6e9ba76fd8b11b6a582abf64e"
server_key_version: "kuduclusterkey@0"
I20260925 05:12:43.220418 16170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:43.241241 16170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:43.243721 16170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:43.243942 16170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:43.244197 16170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:43.244628 16170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:43.244688 16170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:43.244729 16170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:43.244748 16170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:43.251658 16170 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:46269
I20260925 05:12:43.252390 16170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:12:43.252916 16300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:46269 every 8 connection(s)
I20260925 05:12:43.258109 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16170
Sep 25 05:12:43 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:43.263278 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.253318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:55391 (local address 127.3.6.190:44377)
0925 05:12:43.253365 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:43.253367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:43.253690 (+ 323us) server_negotiation.cc:407] Connection header received
0925 05:12:43.254238 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:43.254240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:43.254254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:43.254321 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:43.254865 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.255594 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.256761 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.257073 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.257161 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.259658 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:43.259660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:43.259660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:43.259668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:43.261533 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.262049 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.262051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.262051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.262115 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.262462 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.262463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.262463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.262603 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:43.262681 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:43.263059 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 05:12:43.263163 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:12:43.263477 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.253359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44377 (local address 127.3.6.131:55391)
0925 05:12:43.253524 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:43.253560 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:43.254105 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:43.254383 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:43.254387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:43.254567 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:43.254764 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.254768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.255715 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.255716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.256661 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:43.256663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.257191 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.257192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:43.257301 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.257885 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:43.257894 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:43.259575 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:43.261625 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.261634 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.261636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.261958 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.262206 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:43.262208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:43.262208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:43.262380 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:43.262693 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:43.262700 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:43.262901 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 05:12:43.263034 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 05:12:43.263222 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260925 05:12:43.263989 16301 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44377
I20260925 05:12:43.264153 16301 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:43.264398 16301 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
W20260925 05:12:43.265089 16301 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:44377 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:12:43.265164 16301 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:43.265260 16301 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:43.265532 16301 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:43.265599 16301 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
W20260925 05:12:43.265829 16301 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:12:43.266994 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:43.258583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.3.6.190:44377)
0925 05:12:43.258746 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:43.258748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:43.259672 (+ 924us) server_negotiation.cc:407] Connection header received
0925 05:12:43.259696 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:43.259696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:43.259707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:43.259813 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:43.260594 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.261208 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.262367 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:43.262667 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:43.262736 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:43.264225 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:43.264227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:43.264227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:43.264234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:43.265844 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.266297 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.266297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.266298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.266351 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:43.266535 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:43.266536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:43.266536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:43.266629 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:43.266741 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:43.266850 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 05:12:43.266916 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
Sep 25 05:12:43 dist-test-slave-1lvv krb5kdc[14839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:43.891328 16033 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:43.891512 16033 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:43.892064 15835 ts_manager.cc:194] Registered new tserver with Master: d030b2676d684af3a0707593a8005782 (127.3.6.129:45095)
I20260925 05:12:43.893112 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:50469
I20260925 05:12:44.086905 16167 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:44.087090 16167 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:44.087381 15835 ts_manager.cc:194] Registered new tserver with Master: f0ec450ae9604d5a8a68bf81df950bcb (127.3.6.130:35757)
I20260925 05:12:44.087853 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:49095
Sep 25 05:12:44 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:44.266019 16301 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:44.266182 16301 heartbeater.cc:511] Master 127.3.6.190:44377 requested a full tablet report, sending...
I20260925 05:12:44.266570 15835 ts_manager.cc:194] Registered new tserver with Master: 752b6698f1594a9c92a65dd440644d54 (127.3.6.131:46269)
I20260925 05:12:44.267083 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:55391
I20260925 05:12:44.284768 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:12:44.294929 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.285840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.3.6.190:44377)
0925 05:12:44.287900 (+ 2060us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.287902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.287911 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:12:44.287935 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.287936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.287947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.288016 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:44.288767 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.289274 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.290365 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.290556 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.290606 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.292114 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:44.292115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:44.292115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:44.292122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:44.293804 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.294205 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.294206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.294206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.294264 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:44.294458 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:44.294459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:44.294459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:44.294570 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:44.294648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.294775 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.294844 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2037,"thread_start_us":73,"threads_started":1}
I20260925 05:12:44.297034 15835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:12:44.299050 15835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:12:44.305932 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.302162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.3.6.129:45095)
0925 05:12:44.302328 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.302334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.302657 (+ 323us) server_negotiation.cc:407] Connection header received
0925 05:12:44.302776 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.302779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.302798 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.302915 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:44.303138 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.303894 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.305011 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.305529 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.305621 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.305687 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.305805 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.305849 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260925 05:12:44.305930 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.302058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:45095 (local address 127.0.0.1:34782)
0925 05:12:44.302519 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.302564 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.302652 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.302989 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.302992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.302995 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:44.303071 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.303072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.303981 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.303983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.304918 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.304921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.305621 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.305622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.305702 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.305716 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.305791 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.305847 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":86,"threads_started":1}
I20260925 05:12:44.306875 15948 tablet_service.cc:1467] Processing CreateTablet for tablet d0b628ce7a114398b7d300f90cb46db6 (DEFAULT_TABLE table=test-table [id=f75662a9a30f4202b0d4cd589d898ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:44.307300 15948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b628ce7a114398b7d300f90cb46db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:44.308279 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.302091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:35757 (local address 127.0.0.1:50952)
0925 05:12:44.303957 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.303968 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.304062 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.304362 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.304363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.304365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:44.304411 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.304413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.305747 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.305748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.307063 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.307065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.308058 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.308058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.308122 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.308128 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.308182 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.308227 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1849,"thread_start_us":79,"threads_started":1}
I20260925 05:12:44.308456 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.302489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.3.6.130:35757)
0925 05:12:44.302859 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.302864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.304031 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 05:12:44.304152 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.304155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.304174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.304231 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:44.304491 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.305632 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.307172 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.307964 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.308055 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.308109 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.308361 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.308396 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":57,"threads_started":1}
I20260925 05:12:44.309093 16101 tablet_service.cc:1467] Processing CreateTablet for tablet d0b628ce7a114398b7d300f90cb46db6 (DEFAULT_TABLE table=test-table [id=f75662a9a30f4202b0d4cd589d898ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:44.309417 16101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b628ce7a114398b7d300f90cb46db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:44.310294 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.302405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.3.6.131:46269)
0925 05:12:44.303894 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.303900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.306086 (+ 2186us) server_negotiation.cc:407] Connection header received
0925 05:12:44.306209 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.306212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.306230 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.306287 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:44.306523 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.307599 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.308942 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.309673 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.309756 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.309813 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.310144 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.310188 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":58,"threads_started":1}
I20260925 05:12:44.310411 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.302283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:46269 (local address 127.0.0.1:45974)
0925 05:12:44.306010 (+ 3727us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.306022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.306118 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.306403 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.306404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.306406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:44.306440 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.306441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.307690 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.307691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.308845 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.308849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.309786 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.309787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.309922 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.309927 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.309987 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.310028 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3709,"thread_start_us":73,"threads_started":1}
I20260925 05:12:44.311112 16229 tablet_service.cc:1467] Processing CreateTablet for tablet d0b628ce7a114398b7d300f90cb46db6 (DEFAULT_TABLE table=test-table [id=f75662a9a30f4202b0d4cd589d898ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:44.311441 16229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b628ce7a114398b7d300f90cb46db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:44.311929 16334 tablet_bootstrap.cc:492] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782: Bootstrap starting.
I20260925 05:12:44.312662 16334 tablet_bootstrap.cc:654] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:44.313197 16334 log.cc:824] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:44.313308 16335 tablet_bootstrap.cc:492] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Bootstrap starting.
I20260925 05:12:44.314029 16335 tablet_bootstrap.cc:654] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:44.314350 16335 log.cc:824] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:44.315294 16338 tablet_bootstrap.cc:492] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Bootstrap starting.
I20260925 05:12:44.315953 16338 tablet_bootstrap.cc:654] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:44.316298 16338 log.cc:824] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:44.317108 16338 tablet_bootstrap.cc:492] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: No bootstrap required, opened a new log
I20260925 05:12:44.317180 16338 ts_tablet_manager.cc:1397] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:12:44.319370 16338 raft_consensus.cc:348] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.319543 16338 raft_consensus.cc:374] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:44.319590 16338 raft_consensus.cc:729] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752b6698f1594a9c92a65dd440644d54, State: Initialized, Role: FOLLOWER
I20260925 05:12:44.319706 16338 consensus_queue.cc:260] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.319990 16338 ts_tablet_manager.cc:1428] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:44.320194 16301 heartbeater.cc:503] Master 127.3.6.190:44377 was elected leader, sending a full tablet report...
I20260925 05:12:44.324484 16334 tablet_bootstrap.cc:492] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782: No bootstrap required, opened a new log
I20260925 05:12:44.324570 16334 ts_tablet_manager.cc:1397] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260925 05:12:44.324926 16335 tablet_bootstrap.cc:492] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: No bootstrap required, opened a new log
I20260925 05:12:44.324980 16335 ts_tablet_manager.cc:1397] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260925 05:12:44.326776 16334 raft_consensus.cc:348] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.326957 16334 raft_consensus.cc:374] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:44.326989 16334 raft_consensus.cc:729] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d030b2676d684af3a0707593a8005782, State: Initialized, Role: FOLLOWER
I20260925 05:12:44.327107 16334 consensus_queue.cc:260] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.327418 16033 heartbeater.cc:503] Master 127.3.6.190:44377 was elected leader, sending a full tablet report...
I20260925 05:12:44.328150 16334 ts_tablet_manager.cc:1428] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 05:12:44.329380 16335 raft_consensus.cc:348] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.329553 16335 raft_consensus.cc:374] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:44.329619 16335 raft_consensus.cc:729] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ec450ae9604d5a8a68bf81df950bcb, State: Initialized, Role: FOLLOWER
I20260925 05:12:44.329780 16335 consensus_queue.cc:260] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.330102 16167 heartbeater.cc:503] Master 127.3.6.190:44377 was elected leader, sending a full tablet report...
I20260925 05:12:44.330102 16335 ts_tablet_manager.cc:1428] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20260925 05:12:44.370692 16035 tablet.cc:2369] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:44.440189 16340 raft_consensus.cc:482] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:44.440366 16340 raft_consensus.cc:504] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.440820 16340 leader_election.cc:288] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d030b2676d684af3a0707593a8005782 (127.3.6.129:45095), f0ec450ae9604d5a8a68bf81df950bcb (127.3.6.130:35757)
I20260925 05:12:44.446481 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.441394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:33475 (local address 127.3.6.129:45095)
0925 05:12:44.441455 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.441457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.442156 (+ 699us) server_negotiation.cc:407] Connection header received
0925 05:12:44.442269 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.442271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.442284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.442315 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:44.442575 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.443312 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.445018 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.445798 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.445853 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.445867 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.446290 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.446356 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:12:44.447273 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.441147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:45095 (local address 127.3.6.131:33475)
0925 05:12:44.442076 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.442089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.442186 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.442395 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.442396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.442398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:44.442491 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.442493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.443519 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.443521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.444874 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.444879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.446084 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.446086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.446185 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.446191 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.447047 (+ 856us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.447117 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":107,"threads_started":1}
I20260925 05:12:44.447567 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.441957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:42089 (local address 127.3.6.130:35757)
0925 05:12:44.442010 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.442011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.444182 (+ 2171us) server_negotiation.cc:407] Connection header received
0925 05:12:44.444284 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.444286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.444299 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.444328 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:44.444545 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.445219 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.446590 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.447164 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.447214 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.447224 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.447454 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.447495 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:12:44.447917 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.441760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:35757 (local address 127.3.6.131:42089)
0925 05:12:44.444100 (+ 2340us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.444114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:44.444212 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:44.444405 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:44.444406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:44.444409 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:44.444474 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.444476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.445328 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.445329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.446475 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:44.446477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.447285 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.447286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:44.447373 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.447376 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:12:44.447792 (+ 416us) client_negotiation.cc:240] Negotiation successful
0925 05:12:44.447836 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2318,"thread_start_us":91,"threads_started":1}
I20260925 05:12:44.447893 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b628ce7a114398b7d300f90cb46db6" candidate_uuid: "752b6698f1594a9c92a65dd440644d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d030b2676d684af3a0707593a8005782" is_pre_election: true
I20260925 05:12:44.448128 15987 raft_consensus.cc:2454] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752b6698f1594a9c92a65dd440644d54 in term 0.
I20260925 05:12:44.448447 16188 leader_election.cc:302] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752b6698f1594a9c92a65dd440644d54, d030b2676d684af3a0707593a8005782; no voters:
I20260925 05:12:44.448589 16121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b628ce7a114398b7d300f90cb46db6" candidate_uuid: "752b6698f1594a9c92a65dd440644d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" is_pre_election: true
I20260925 05:12:44.448771 16121 raft_consensus.cc:2454] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752b6698f1594a9c92a65dd440644d54 in term 0.
I20260925 05:12:44.448907 16340 raft_consensus.cc:2781] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:12:44.448984 16340 raft_consensus.cc:482] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:12:44.449018 16340 raft_consensus.cc:3037] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:44.450248 16340 raft_consensus.cc:504] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.450446 16340 leader_election.cc:288] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [CANDIDATE]: Term 1 election: Requested vote from peers d030b2676d684af3a0707593a8005782 (127.3.6.129:45095), f0ec450ae9604d5a8a68bf81df950bcb (127.3.6.130:35757)
I20260925 05:12:44.450760 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b628ce7a114398b7d300f90cb46db6" candidate_uuid: "752b6698f1594a9c92a65dd440644d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d030b2676d684af3a0707593a8005782"
I20260925 05:12:44.450853 15987 raft_consensus.cc:3037] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:44.451020 16121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b628ce7a114398b7d300f90cb46db6" candidate_uuid: "752b6698f1594a9c92a65dd440644d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ec450ae9604d5a8a68bf81df950bcb"
I20260925 05:12:44.451107 16121 raft_consensus.cc:3037] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:44.452057 16121 raft_consensus.cc:2454] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752b6698f1594a9c92a65dd440644d54 in term 1.
I20260925 05:12:44.452252 16191 leader_election.cc:302] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752b6698f1594a9c92a65dd440644d54, f0ec450ae9604d5a8a68bf81df950bcb; no voters:
I20260925 05:12:44.452266 15987 raft_consensus.cc:2454] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752b6698f1594a9c92a65dd440644d54 in term 1.
I20260925 05:12:44.452440 16340 raft_consensus.cc:2781] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:44.452616 16340 raft_consensus.cc:686] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 1 LEADER]: Becoming Leader. State: Replica: 752b6698f1594a9c92a65dd440644d54, State: Running, Role: LEADER
I20260925 05:12:44.452713 16340 consensus_queue.cc:237] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } }
I20260925 05:12:44.453529 15834 catalog_manager.cc:5495] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 752b6698f1594a9c92a65dd440644d54 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "752b6698f1594a9c92a65dd440644d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "752b6698f1594a9c92a65dd440644d54" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 46269 } health_report { overall_health: HEALTHY } } }
Sep 25 05:12:44 dist-test-slave-1lvv krb5kdc[14839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for HTTP/127.3.6.148@KRBTEST.COM
I20260925 05:12:44.494174 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.490669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.3.6.131:46269)
0925 05:12:44.490732 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:44.490734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:44.490742 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:12:44.490787 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:44.490787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:44.490830 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:44.490902 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:44.491120 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.491723 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.493015 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:44.493291 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:44.493362 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:44.493583 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:44.493726 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:44.493804 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:44.494066 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:12:44.494089 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:12:44.496595 16121 raft_consensus.cc:1261] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 1 FOLLOWER]: Refusing update from remote peer 752b6698f1594a9c92a65dd440644d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:44.497068 16340 consensus_queue.cc:1035] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ec450ae9604d5a8a68bf81df950bcb" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:44.498986 15987 raft_consensus.cc:1261] T d0b628ce7a114398b7d300f90cb46db6 P d030b2676d684af3a0707593a8005782 [term 1 FOLLOWER]: Refusing update from remote peer 752b6698f1594a9c92a65dd440644d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:44.499359 16340 consensus_queue.cc:1035] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d030b2676d684af3a0707593a8005782" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:12:44.503940 16302 tablet.cc:2369] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:44.507313 16350 mvcc.cc:205] Tried to move back new op lower bound from 7333122721775284224 to 7333122721598898176. Current Snapshot: MvccSnapshot[applied={T|T < 7333122721775284224}]
I20260925 05:12:44.507817 16349 mvcc.cc:205] Tried to move back new op lower bound from 7333122721775284224 to 7333122721598898176. Current Snapshot: MvccSnapshot[applied={T|T < 7333122721775284224}]
I20260925 05:12:44.508030 16352 mvcc.cc:205] Tried to move back new op lower bound from 7333122721775284224 to 7333122721598898176. Current Snapshot: MvccSnapshot[applied={T|T < 7333122721775284224}]
I20260925 05:12:44.510944 15834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:44.511071 15834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:44.511912 15834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e53e2426376142b89c544aacb4f3f4d9: Sending DeleteTablet for 3 replicas of tablet d0b628ce7a114398b7d300f90cb46db6
I20260925 05:12:44.512320 16101 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b628ce7a114398b7d300f90cb46db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:44 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:50952
I20260925 05:12:44.512459 15948 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b628ce7a114398b7d300f90cb46db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:44 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:34782
I20260925 05:12:44.512528 16229 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b628ce7a114398b7d300f90cb46db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:44 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45974
I20260925 05:12:44.513094 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 15873
I20260925 05:12:44.519897 16359 tablet_replica.cc:330] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: stopping tablet replica
I20260925 05:12:44.520123 16359 raft_consensus.cc:2229] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:44.520370 16359 raft_consensus.cc:2258] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:44.531921 16361 tablet_replica.cc:330] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: stopping tablet replica
W20260925 05:12:44.532145 15819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:45095 (error 108)
I20260925 05:12:44.532158 16359 ts_tablet_manager.cc:1905] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:44.532196 16361 raft_consensus.cc:2229] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:12:44.532359 16361 raft_consensus.cc:2258] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:44.532812 16361 ts_tablet_manager.cc:1905] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:44.534312 16359 ts_tablet_manager.cc:1918] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:44.534422 16359 log.cc:1196] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/d0b628ce7a114398b7d300f90cb46db6
I20260925 05:12:44.534456 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16034
I20260925 05:12:44.534623 16361 ts_tablet_manager.cc:1918] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:44.534686 16361 log.cc:1196] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/d0b628ce7a114398b7d300f90cb46db6
I20260925 05:12:44.534696 16359 ts_tablet_manager.cc:1939] T d0b628ce7a114398b7d300f90cb46db6 P f0ec450ae9604d5a8a68bf81df950bcb: Deleting consensus metadata
I20260925 05:12:44.534957 16361 ts_tablet_manager.cc:1939] T d0b628ce7a114398b7d300f90cb46db6 P 752b6698f1594a9c92a65dd440644d54: Deleting consensus metadata
I20260925 05:12:44.536615 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.536441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:45095 (local address 127.0.0.1:34798)
0925 05:12:44.536523 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.536569 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:45095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:12:44.536755 15819 catalog_manager.cc:4584] TS d030b2676d684af3a0707593a8005782 (127.3.6.129:45095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0b628ce7a114398b7d300f90cb46db6: Network error: Client connection negotiation failed: client connection to 127.3.6.129:45095: connect: Connection refused (error 111)
I20260925 05:12:44.538666 15822 catalog_manager.cc:4841] TS 752b6698f1594a9c92a65dd440644d54 (127.3.6.131:46269): tablet d0b628ce7a114398b7d300f90cb46db6 (table test-table [id=f75662a9a30f4202b0d4cd589d898ff0]) successfully deleted
I20260925 05:12:44.541474 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16170
I20260925 05:12:44.543711 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:44.543584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:35757 (local address 127.0.0.1:50956)
0925 05:12:44.543638 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:44.543667 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.130:35757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:12:44.547186 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 15802
I20260925 05:12:44.555241 3098 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:12:44.912920 3098 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:12:44.913031 3098 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:12:45.258000 3098 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:12:45.258 UTC [14869] LOG: received smart shutdown request
2026-09-25 05:12:45.262 UTC [14869] LOG: background worker "logical replication launcher" (PID 14878) exited with exit code 1
2026-09-25 05:12:45.262 UTC [14873] LOG: shutting down
2026-09-25 05:12:45.446 UTC [14869] LOG: database system is shut down
2026-09-25T05:12:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49146 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:45 dist-test-slave-1lvv krb5kdc[16371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:45 dist-test-slave-1lvv krb5kdc[16371](info): set up 2 sockets
Sep 25 05:12:45 dist-test-slave-1lvv krb5kdc[16371](info): commencing operation
krb5kdc: starting...
W20260925 05:12:46.611984 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:12:46 dist-test-slave-1lvv krb5kdc[16371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:12:46.742525 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33531
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:33531,127.3.6.189:38863,127.3.6.188:46497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:46.831043 16384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:46.831357 16384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:46.831403 16384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:46.832759 16384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:46.832858 16384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:46.832882 16384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:46.832904 16384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:46.832933 16384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:46.834692 16384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:33531,127.3.6.189:38863,127.3.6.188:46497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:46.835028 16384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:46.835289 16384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:46.840430 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:46.840610 16384 server_base.cc:1034] running on GCE node
W20260925 05:12:46.840451 16389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:46.840453 16390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:46.840967 16384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:46.841332 16384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:46.842521 16384 hybrid_clock.cc:648] HybridClock initialized: now 1790313166842498 us; error 35 us; skew 500 ppm
Sep 25 05:12:46 dist-test-slave-1lvv krb5kdc[16371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:46.850167 16384 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:12:46.850706 16384 webserver.cc:460] Webserver started at http://127.3.6.190:33351/ using document root <none> and password file <none>
I20260925 05:12:46.850977 16384 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:46.851051 16384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:46.851214 16384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:46.852458 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "a894243f25a04abb94c6111989f4de56"
format_stamp: "Formatted at 2026-09-25 05:12:46 on dist-test-slave-1lvv"
server_key: "cd5cd7940ccd38d06afccce1cf37a8d0"
server_key_iv: "8732870b9f8f7c57b2040fe6af61420b"
server_key_version: "encryptionkey@0"
I20260925 05:12:46.852849 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "a894243f25a04abb94c6111989f4de56"
format_stamp: "Formatted at 2026-09-25 05:12:46 on dist-test-slave-1lvv"
server_key: "cd5cd7940ccd38d06afccce1cf37a8d0"
server_key_iv: "8732870b9f8f7c57b2040fe6af61420b"
server_key_version: "encryptionkey@0"
I20260925 05:12:46.854321 16384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:12:46.855156 16399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:46.855333 16384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:46.855460 16384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "a894243f25a04abb94c6111989f4de56"
format_stamp: "Formatted at 2026-09-25 05:12:46 on dist-test-slave-1lvv"
server_key: "cd5cd7940ccd38d06afccce1cf37a8d0"
server_key_iv: "8732870b9f8f7c57b2040fe6af61420b"
server_key_version: "encryptionkey@0"
I20260925 05:12:46.855541 16384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:46.872272 16384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:46.874766 16384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:46.874951 16384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:46.879542 16384 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:33531
I20260925 05:12:46.879586 16451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:33531 every 8 connection(s)
I20260925 05:12:46.880052 16384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:46.881034 16452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:46.882709 16452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:46.888404 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16384
I20260925 05:12:46.888562 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:12:46.888775 3098 external_mini_cluster.cc:1428] Setting key e776fdbe26e712fa40d6e6cbe51d82fa
Sep 25 05:12:46 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:46.893014 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:46.883496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33531 (local address 127.0.0.1:39364)
0925 05:12:46.883698 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:46.883734 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:46.884256 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:46.884554 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:46.884557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:46.884821 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:46.884987 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:46.884991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:46.885996 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:46.885997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:46.886690 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:46.886692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:46.887175 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:46.887176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:46.887234 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:46.887806 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:46.887818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:46.889466 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:46.891368 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:46.891376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:46.891378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:46.891584 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:46.891918 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:46.891919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:46.891919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:46.892007 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:46.892376 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:46.892381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:46.892446 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 05:12:46.892560 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:12:46.892805 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":99,"spinlock_wait_cycles":8448,"thread_start_us":100,"threads_started":1}
I20260925 05:12:46.893014 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:46.883506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.3.6.190:33531)
0925 05:12:46.883729 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:46.883746 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:46.883859 (+ 113us) server_negotiation.cc:407] Connection header received
0925 05:12:46.884419 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:46.884421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:46.884438 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:46.884524 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:46.885075 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:46.885898 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:46.886764 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:46.887099 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:46.887177 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:46.889572 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:46.889580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:46.889580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:46.889587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:46.891199 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:46.891683 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:46.891686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:46.891686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:46.891824 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:46.892074 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:46.892075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:46.892075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:46.892301 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:46.892408 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:46.892585 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 05:12:46.892759 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260925 05:12:46.893828 16452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:46.894215 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:46.894103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49208)
0925 05:12:46.894149 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:46.894192 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:12:46.894394 16403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
I20260925 05:12:46.894843 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:46.894772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49216)
0925 05:12:46.894800 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:46.894825 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 05:12:46.894975 16452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:38863: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.6.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab.
Entry for principal kudu/127.3.6.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab.
I20260925 05:12:46.918701 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:38863
--webserver_interface=127.3.6.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab
--principal=kudu/127.3.6.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:33531,127.3.6.189:38863,127.3.6.188:46497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:46.944231 16452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } attempt: 1
I20260925 05:12:46.944814 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:46.944654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49226)
0925 05:12:46.944732 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:46.944780 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 05:12:46.945205 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:46.945116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49242)
0925 05:12:46.945160 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:46.945187 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:12:46.945344 16452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:38863: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
W20260925 05:12:47.006280 16459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:47.006515 16459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:47.006538 16459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:47.007746 16459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:47.007862 16459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:47.007879 16459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:47.007915 16459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:47.007941 16459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:47.009511 16459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:33531,127.3.6.189:38863,127.3.6.188:46497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab
--principal=kudu/127.3.6.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.6.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:47.009794 16459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:47.010037 16459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:12:47.013538 16452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } attempt: 2
I20260925 05:12:47.014075 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.013925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49252)
0925 05:12:47.013994 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.014032 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:12:47.014506 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.014372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49254)
0925 05:12:47.014443 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.014484 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 05:12:47.014636 16452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:38863: Network error: Client connection negotiation failed: client connection to 127.3.6.189:38863: connect: Connection refused (error 111)
W20260925 05:12:47.015281 16467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:47.015297 16465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:47.015277 16464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:47.015651 16459 server_base.cc:1034] running on GCE node
I20260925 05:12:47.015964 16459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:47.016300 16459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:47.017494 16459 hybrid_clock.cc:648] HybridClock initialized: now 1790313167017433 us; error 80 us; skew 500 ppm
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:47.020274 16459 init.cc:377] Logged in from keytab as kudu/127.3.6.189@KRBTEST.COM (short username kudu)
I20260925 05:12:47.020743 16459 webserver.cc:460] Webserver started at http://127.3.6.189:46343/ using document root <none> and password file <none>
I20260925 05:12:47.021003 16459 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:47.021073 16459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:47.021224 16459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:47.022413 16459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data/instance:
uuid: "88b4ef49da0842438f1176ecac3c3b8e"
format_stamp: "Formatted at 2026-09-25 05:12:47 on dist-test-slave-1lvv"
server_key: "e37f5f8cadf240ba638ded52a55b4799"
server_key_iv: "d384b2a3ae88e3bd55c4352ee63c71ee"
server_key_version: "encryptionkey@0"
I20260925 05:12:47.022806 16459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/wal/instance:
uuid: "88b4ef49da0842438f1176ecac3c3b8e"
format_stamp: "Formatted at 2026-09-25 05:12:47 on dist-test-slave-1lvv"
server_key: "e37f5f8cadf240ba638ded52a55b4799"
server_key_iv: "d384b2a3ae88e3bd55c4352ee63c71ee"
server_key_version: "encryptionkey@0"
I20260925 05:12:47.024323 16459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:12:47.025084 16474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:47.025266 16459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:12:47.025378 16459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/wal
uuid: "88b4ef49da0842438f1176ecac3c3b8e"
format_stamp: "Formatted at 2026-09-25 05:12:47 on dist-test-slave-1lvv"
server_key: "e37f5f8cadf240ba638ded52a55b4799"
server_key_iv: "d384b2a3ae88e3bd55c4352ee63c71ee"
server_key_version: "encryptionkey@0"
I20260925 05:12:47.025454 16459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:47.049312 16459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:47.051522 16459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:47.051710 16459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:47.055862 16459 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.189:38863
I20260925 05:12:47.055938 16526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.189:38863 every 8 connection(s)
I20260925 05:12:47.056296 16459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
I20260925 05:12:47.057197 16527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:47.058846 16527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:47.064617 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16459
I20260925 05:12:47.064764 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-1/wal/instance
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:47.064990 3098 external_mini_cluster.cc:1428] Setting key c95575a687d86a9049a7c7788f716db3
I20260925 05:12:47.069012 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.059595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33531 (local address 127.0.0.1:39366)
0925 05:12:47.059747 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.059831 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.060274 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.060503 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.060506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.060764 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.060963 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.060967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.061642 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.061642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.062500 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.062503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.062870 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.062871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.062984 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.063489 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.063507 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.065241 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.067229 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.067239 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.067242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.067649 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.067945 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.067946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.067947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.068088 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.068455 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.068462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.068598 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.068694 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.068833 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260925 05:12:47.069008 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.059588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.3.6.190:33531)
0925 05:12:47.059638 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.059640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.059927 (+ 287us) server_negotiation.cc:407] Connection header received
0925 05:12:47.060382 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.060383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.060396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.060451 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.061036 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.061538 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.062585 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.062794 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.062848 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.065355 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.065357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.065357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.065367 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.067132 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.067747 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.067748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.067749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.067870 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.068188 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.068189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.068189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.068353 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.068429 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.068738 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.068847 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:12:47.069787 16527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.6.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.188@KRBTEST.COM" created.
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:12:47.080924 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.070101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49266)
0925 05:12:47.070360 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.070371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.070423 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.070573 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.070574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.070727 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.070783 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.070785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.071686 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.071687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.072249 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.072250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.072773 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.072774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.072845 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.073324 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.073330 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.077352 (+ 4022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.079665 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.079666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.079667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.079931 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.080168 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.080169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.080169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.080255 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.080643 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.080643 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.080685 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.080733 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.080827 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 05:12:47.081113 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.070188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.3.6.189:38863)
0925 05:12:47.070349 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.070354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.070422 (+ 68us) server_negotiation.cc:407] Connection header received
0925 05:12:47.070482 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.070485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.070499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.070572 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.070875 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.071584 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.072357 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.072681 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.072774 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.077487 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.077506 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.077506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.077512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.079554 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.080018 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.080023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.080023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.080095 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.080330 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.080331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.080331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.080575 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.080664 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.080778 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.080911 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":128,"thread_start_us":124,"threads_started":1}
I20260925 05:12:47.081842 16527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:47.082233 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.082097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59700)
0925 05:12:47.082149 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.082207 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 05:12:47.082424 16475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
I20260925 05:12:47.082906 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.082820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59706)
0925 05:12:47.082864 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.082889 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:12:47.083043 16527 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:46497: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab.
Entry for principal kudu/127.3.6.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab.
I20260925 05:12:47.094803 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:46497
--webserver_interface=127.3.6.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab
--principal=kudu/127.3.6.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:33531,127.3.6.189:38863,127.3.6.188:46497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:47.105808 16452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } attempt: 3
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:12:47.114022 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.106271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49278)
0925 05:12:47.106385 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.106399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.106450 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.106622 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.106623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.106782 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.106850 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.106852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.107884 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.107885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.108420 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.108422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.108946 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.108947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.109018 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.109544 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.109550 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.110959 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.112925 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.112926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.112926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.113123 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.113336 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.113337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.113337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.113396 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.113701 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.113702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.113755 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.113840 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.113947 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 05:12:47.114075 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.106355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.3.6.189:38863)
0925 05:12:47.106423 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.106425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.106441 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:12:47.106528 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.106529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.106544 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.106602 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.106932 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.107750 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.108508 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.108861 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.108940 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.111055 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.111057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.111057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.111065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.112816 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.113203 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.113204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.113204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.113269 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.113462 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.113463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.113463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.113603 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.113676 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.113886 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.113978 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:12:47.114687 16452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:47.115139 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.115002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59720)
0925 05:12:47.115061 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.115100 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 05:12:47.115476 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.115381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59724)
0925 05:12:47.115427 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.115454 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:12:47.115572 16452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:46497: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
I20260925 05:12:47.132293 16527 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } attempt: 1
I20260925 05:12:47.132802 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.132654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59738)
0925 05:12:47.132720 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.132761 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 05:12:47.133195 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.133093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59754)
0925 05:12:47.133143 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.133171 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 05:12:47.133287 16527 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:46497: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
I20260925 05:12:47.146727 16452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } attempt: 1
I20260925 05:12:47.147235 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.147097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59770)
0925 05:12:47.147161 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.147204 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:12:47.147609 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.147508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59784)
0925 05:12:47.147556 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.147585 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 05:12:47.147702 16452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:46497: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
W20260925 05:12:47.183097 16534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:47.183349 16534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:47.183384 16534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:47.184661 16534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:47.184712 16534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:47.184726 16534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:47.184741 16534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:47.184753 16534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:47.186448 16534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:33531,127.3.6.189:38863,127.3.6.188:46497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab
--principal=kudu/127.3.6.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:46497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.6.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:47.186746 16534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:47.187026 16534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:47.192245 16539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:47.192238 16542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:47.192236 16540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:47.192267 16534 server_base.cc:1034] running on GCE node
I20260925 05:12:47.192762 16534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:47.193094 16534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:47.194247 16534 hybrid_clock.cc:648] HybridClock initialized: now 1790313167194220 us; error 45 us; skew 500 ppm
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:47.196919 16534 init.cc:377] Logged in from keytab as kudu/127.3.6.188@KRBTEST.COM (short username kudu)
I20260925 05:12:47.197417 16534 webserver.cc:460] Webserver started at http://127.3.6.188:41395/ using document root <none> and password file <none>
I20260925 05:12:47.197618 16534 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:47.197661 16534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:47.197759 16534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:47.198896 16534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data/instance:
uuid: "f08a986ce38748ce87340299a3ff95eb"
format_stamp: "Formatted at 2026-09-25 05:12:47 on dist-test-slave-1lvv"
server_key: "ac50917058b377688822ecec79d71a18"
server_key_iv: "6ad166678d9b5ea73df2483faa79c2f2"
server_key_version: "encryptionkey@0"
I20260925 05:12:47.199265 16534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/wal/instance:
uuid: "f08a986ce38748ce87340299a3ff95eb"
format_stamp: "Formatted at 2026-09-25 05:12:47 on dist-test-slave-1lvv"
server_key: "ac50917058b377688822ecec79d71a18"
server_key_iv: "6ad166678d9b5ea73df2483faa79c2f2"
server_key_version: "encryptionkey@0"
I20260925 05:12:47.200795 16534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:12:47.201427 16527 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } attempt: 2
I20260925 05:12:47.201582 16549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:47.201762 16534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:12:47.201889 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.201729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59800)
0925 05:12:47.201797 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.201842 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 05:12:47.201864 16534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/wal
uuid: "f08a986ce38748ce87340299a3ff95eb"
format_stamp: "Formatted at 2026-09-25 05:12:47 on dist-test-slave-1lvv"
server_key: "ac50917058b377688822ecec79d71a18"
server_key_iv: "6ad166678d9b5ea73df2483faa79c2f2"
server_key_version: "encryptionkey@0"
I20260925 05:12:47.201938 16534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:47.202252 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.202153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59806)
0925 05:12:47.202198 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.202227 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:12:47.202382 16527 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:46497: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
I20260925 05:12:47.222857 16452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } attempt: 2
I20260925 05:12:47.223398 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.223228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59818)
0925 05:12:47.223321 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.223367 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 05:12:47.223824 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.223666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59826)
0925 05:12:47.223719 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.223751 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 05:12:47.223919 16452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:46497: Network error: Client connection negotiation failed: client connection to 127.3.6.188:46497: connect: Connection refused (error 111)
I20260925 05:12:47.238847 16534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:47.240950 16534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:47.241135 16534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:47.245458 16534 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.188:46497
I20260925 05:12:47.245494 16601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.188:46497 every 8 connection(s)
I20260925 05:12:47.245895 16534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
I20260925 05:12:47.246845 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:47.248632 16602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:47.251101 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16534
I20260925 05:12:47.251250 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790313008371521-3098-0/minicluster-data/master-2/wal/instance
I20260925 05:12:47.251446 3098 external_mini_cluster.cc:1428] Setting key 867abb5a72995d42a208c6c653fd3032
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:12:47.259035 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.249342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33531 (local address 127.0.0.1:39382)
0925 05:12:47.249514 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.249556 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.250016 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.250299 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.250302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.250576 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.250777 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.250781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.251625 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.251627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.252608 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.252611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.253094 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.253095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.253233 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.253752 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.253762 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.255242 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.257170 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.257177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.257179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.257472 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.257694 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.257694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.257694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.257902 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.258220 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.258241 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.258424 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.258557 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.258720 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260925 05:12:47.259195 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.249426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.3.6.190:33531)
0925 05:12:47.249500 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.249502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.249673 (+ 171us) server_negotiation.cc:407] Connection header received
0925 05:12:47.250137 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.250139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.250150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.250219 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.250859 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.251528 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.252699 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.253010 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.253080 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.255338 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.255340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.255340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.255348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.257083 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.257557 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.257558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.257559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.257624 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.257976 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.257977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.257977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.258145 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.258216 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.258987 (+ 771us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.259074 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:12:47.259873 16602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:47.261238 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.251936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.3.6.188:46497)
0925 05:12:47.252112 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.252117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.252143 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:12:47.252180 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.252183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.252202 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.252321 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.252982 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.254022 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.254773 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.255141 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.255225 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.257686 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.257694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.257694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.257701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.259734 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.260222 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.260225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.260225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.260320 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.260564 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.260565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.260565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.260733 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.260913 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.260959 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.261038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:47.267138 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.260139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:38863 (local address 127.0.0.1:49282)
0925 05:12:47.260193 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.260206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.260251 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.260426 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.260427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.260584 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.260631 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.260632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.261508 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.261509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.262237 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.262240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.262618 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.262619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.262681 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.263084 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.263088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.264503 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.266002 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.266004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.266004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.266182 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.266426 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.266427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.266427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.266487 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.266782 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.266783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.266833 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.266933 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.267038 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 05:12:47.267138 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.260232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.3.6.189:38863)
0925 05:12:47.260307 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.260309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.260317 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:12:47.260343 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.260344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.260356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.260411 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.260706 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.261422 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.262330 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.262542 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.262623 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.264605 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.264607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.264607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.264615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.265906 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.266289 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.266291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.266292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.266354 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.266561 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.266562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.266562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.266700 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.266792 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.266961 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.267038 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:12:47.267670 16602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:12:47.269773 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.262113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.3.6.190:33531)
0925 05:12:47.262158 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.262160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.262169 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:12:47.262212 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.262213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.262222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.262290 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.262791 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.263354 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.264415 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.264609 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.264696 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.267006 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.267007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.267007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.267015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.268546 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.268958 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.268959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.268959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.269010 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.269221 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.269221 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.269222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.269347 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.269558 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.269595 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.269676 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:12:47.275388 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.267957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59840)
0925 05:12:47.268004 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.268014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.268050 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.268743 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.268744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.268898 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.268943 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.268945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.269549 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.269550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.270278 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.270280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.270695 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.270696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.270745 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.271156 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.271161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.272431 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.274171 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.274178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.274179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.274437 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.274637 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.274638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.274639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.274710 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.274994 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.274995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.275043 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.275128 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.275230 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 05:12:47.275444 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.268008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.3.6.188:46497)
0925 05:12:47.268635 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.268636 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.268642 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:12:47.268657 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.268658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.268668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.268708 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.269005 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.269467 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.270338 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.270608 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.270695 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.272510 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.272511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.272512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.272517 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.274061 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.274526 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.274526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.274527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.274574 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.274774 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.274775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.274775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.274925 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.274979 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.275164 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.275273 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":607}
I20260925 05:12:47.277163 16602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: Bootstrap starting.
I20260925 05:12:47.278082 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.270463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.3.6.189:38863)
0925 05:12:47.270504 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.270506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.270514 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:12:47.270537 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.270538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.270547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.270596 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.271052 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.271498 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.272255 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.272554 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.272618 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.275342 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.275344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.275344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.275351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.276981 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.277369 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.277370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.277370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.277420 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.277632 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.277633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.277633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.277727 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.277892 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.277962 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.278021 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:12:47.278308 16602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:47.278672 16602 log.cc:824] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:47.279693 16602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: No bootstrap required, opened a new log
I20260925 05:12:47.280205 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:47.282156 16602 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.282351 16602 raft_consensus.cc:374] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:47.282406 16602 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08a986ce38748ce87340299a3ff95eb, State: Initialized, Role: FOLLOWER
I20260925 05:12:47.282548 16602 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.282846 16615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.282896 16602 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:47.283041 16615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:12:47.284974 16626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:12:47.285929 16626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:12:47.286593 16534 master_runner.cc:186] Error getting master registration for 127.3.6.190:33531: OK, instance_id { permanent_uuid: "a894243f25a04abb94c6111989f4de56" instance_seqno: 1790313166878053 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:12:47.289588 16534 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:12:47.290012 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.281824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.3.6.189:38863)
0925 05:12:47.281872 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.281873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.282036 (+ 163us) server_negotiation.cc:407] Connection header received
0925 05:12:47.282152 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.282153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.282165 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.282235 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.283006 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.283715 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.284832 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.285160 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.285239 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.286822 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.286823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.286824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.286831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.288619 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.289103 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.289105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.289106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.289169 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.289407 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.289408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.289408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.289544 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.289650 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.289835 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.289914 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:12:47.291903 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.282011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.3.6.190:33531)
0925 05:12:47.282058 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.282060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.282278 (+ 218us) server_negotiation.cc:407] Connection header received
0925 05:12:47.282521 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.282522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.282535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.282600 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.283330 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.283983 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.284996 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.285287 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.285352 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.288214 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.288215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.288215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.288223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.289902 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.290596 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.290597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.290598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.290655 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.291134 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.291135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.291136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.291289 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.291386 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.291710 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.291819 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:12:47.292995 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.282030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.3.6.188:46497)
0925 05:12:47.282094 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.282095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.284459 (+ 2364us) server_negotiation.cc:407] Connection header received
0925 05:12:47.284610 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.284611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.284621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.284650 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.285386 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.286033 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.287084 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.287361 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.287406 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.289735 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.289736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.289736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.289744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.291537 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.292190 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.292192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.292192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.292255 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.292504 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.292505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.292505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.292628 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.292709 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.292878 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.292941 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:12:47.293520 16527 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } attempt: 3
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:12:47.301136 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.293777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59852)
0925 05:12:47.293842 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.293855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.293901 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.294119 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.294121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.294283 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.294332 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.294334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.295028 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.295030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.295905 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.295907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.296361 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.296362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.296401 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.296851 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.296855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.298299 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.300177 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.300178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.300178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.300367 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.300574 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.300575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.300575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.300626 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.300896 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.300897 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.300935 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.300988 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.301067 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 05:12:47.301183 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.293981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.3.6.188:46497)
0925 05:12:47.294017 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.294019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.294025 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:12:47.294040 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.294041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.294048 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.294092 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.294411 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.294938 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.295985 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.296280 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.296362 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.298393 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.298394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.298394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.298403 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.300090 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.300442 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.300443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.300444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.300505 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.300685 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.300686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.300686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.300829 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.300905 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.301011 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.301089 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:12:47.302806 16527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: Bootstrap starting.
I20260925 05:12:47.303467 16527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:47.303808 16527 log.cc:824] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:47.304695 16527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: No bootstrap required, opened a new log
I20260925 05:12:47.306236 16527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.306352 16527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:47.306375 16527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b4ef49da0842438f1176ecac3c3b8e, State: Initialized, Role: FOLLOWER
I20260925 05:12:47.306528 16527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.306790 16629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.306869 16527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:47.306870 16629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:12:47.308802 16640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:12:47.308909 16640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:12:47.308980 16459 master_runner.cc:186] Error getting master registration for 127.3.6.190:33531: OK, instance_id { permanent_uuid: "a894243f25a04abb94c6111989f4de56" instance_seqno: 1790313166878053 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:12:47.311856 16459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:12:47.323082 16452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } attempt: 3
Sep 25 05:12:47 dist-test-slave-1lvv krb5kdc[16371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:12:47.331892 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.323566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.3.6.188:46497)
0925 05:12:47.323642 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:47.323645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:47.323656 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:12:47.323684 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:47.323684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:47.323695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:47.323783 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:47.324199 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.324964 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.325975 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.326294 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.326369 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.328788 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:47.328789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:47.328790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:47.328798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:47.330567 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.330998 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.330999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.331000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.331066 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:47.331291 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:47.331291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:47.331292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:47.331438 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:47.331508 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:47.331679 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 05:12:47.331788 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:12:47.331892 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:47.323479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:46497 (local address 127.0.0.1:59866)
0925 05:12:47.323531 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:47.323552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:47.323617 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:47.323844 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:47.323845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:47.324054 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:47.324111 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.324113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.325076 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.325077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.325890 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:47.325893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:47.326377 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:47.326378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:47.326442 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:47.326980 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:47.326987 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:47.328682 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:47.330663 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.330664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.330665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.330915 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.331141 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:47.331142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:47.331142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:47.331215 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:47.331511 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:47.331512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:47.331565 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 05:12:47.331638 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:12:47.331780 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 05:12:47.333777 16452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56: Bootstrap starting.
I20260925 05:12:47.334555 16452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:47.334882 16452 log.cc:824] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:47.336509 16452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56: No bootstrap required, opened a new log
I20260925 05:12:47.338552 16452 raft_consensus.cc:348] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.338699 16452 raft_consensus.cc:374] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:47.338744 16452 raft_consensus.cc:729] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a894243f25a04abb94c6111989f4de56, State: Initialized, Role: FOLLOWER
I20260925 05:12:47.338873 16452 consensus_queue.cc:260] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.339200 16452 sys_catalog.cc:564] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:47.339171 16643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.339258 16643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:12:47.341055 16654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:12:47.341171 16654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:12:47.343298 16384 master_runner.cc:263] No leader master found from master a894243f25a04abb94c6111989f4de56
I20260925 05:12:47.343346 16384 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 05:12:47.375896 16405 tablet.cc:2369] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:47.426411 16643 raft_consensus.cc:482] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:47.426548 16643 raft_consensus.cc:504] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.426857 16643 leader_election.cc:288] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b4ef49da0842438f1176ecac3c3b8e (127.3.6.189:38863), f08a986ce38748ce87340299a3ff95eb (127.3.6.188:46497)
I20260925 05:12:47.427251 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a894243f25a04abb94c6111989f4de56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08a986ce38748ce87340299a3ff95eb" is_pre_election: true
I20260925 05:12:47.427425 16576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a894243f25a04abb94c6111989f4de56 in term 0.
I20260925 05:12:47.428001 16400 leader_election.cc:302] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a894243f25a04abb94c6111989f4de56, f08a986ce38748ce87340299a3ff95eb; no voters:
I20260925 05:12:47.428261 16643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:12:47.428321 16643 raft_consensus.cc:482] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:12:47.428344 16643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:47.429361 16643 raft_consensus.cc:504] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.429497 16643 leader_election.cc:288] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [CANDIDATE]: Term 1 election: Requested vote from peers 88b4ef49da0842438f1176ecac3c3b8e (127.3.6.189:38863), f08a986ce38748ce87340299a3ff95eb (127.3.6.188:46497)
I20260925 05:12:47.429888 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a894243f25a04abb94c6111989f4de56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08a986ce38748ce87340299a3ff95eb"
I20260925 05:12:47.429973 16576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:47.430787 16576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a894243f25a04abb94c6111989f4de56 in term 1.
I20260925 05:12:47.430995 16400 leader_election.cc:302] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a894243f25a04abb94c6111989f4de56, f08a986ce38748ce87340299a3ff95eb; no voters:
I20260925 05:12:47.431289 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a894243f25a04abb94c6111989f4de56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4ef49da0842438f1176ecac3c3b8e" is_pre_election: true
I20260925 05:12:47.431457 16501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a894243f25a04abb94c6111989f4de56 in term 0.
I20260925 05:12:47.431564 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a894243f25a04abb94c6111989f4de56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4ef49da0842438f1176ecac3c3b8e"
I20260925 05:12:47.431607 16500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:47.432564 16500 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a894243f25a04abb94c6111989f4de56 in term 1.
I20260925 05:12:47.432919 16643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:47.433054 16643 raft_consensus.cc:686] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [term 1 LEADER]: Becoming Leader. State: Replica: a894243f25a04abb94c6111989f4de56, State: Running, Role: LEADER
I20260925 05:12:47.433130 16643 consensus_queue.cc:237] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } }
I20260925 05:12:47.433495 16643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a894243f25a04abb94c6111989f4de56. Latest consensus state: current_term: 1 leader_uuid: "a894243f25a04abb94c6111989f4de56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.433552 16643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:47.434943 16658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:47.435200 16658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:47.437264 16576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [term 1 FOLLOWER]: Refusing update from remote peer a894243f25a04abb94c6111989f4de56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:47.437640 16643 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:47.438977 16615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a894243f25a04abb94c6111989f4de56. Latest consensus state: current_term: 1 leader_uuid: "a894243f25a04abb94c6111989f4de56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.439064 16615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:47.439539 16500 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [term 1 FOLLOWER]: Refusing update from remote peer a894243f25a04abb94c6111989f4de56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:47.439853 16643 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:47.441208 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a894243f25a04abb94c6111989f4de56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.441306 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:47.441561 16629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a894243f25a04abb94c6111989f4de56. Latest consensus state: current_term: 1 leader_uuid: "a894243f25a04abb94c6111989f4de56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.441658 16629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:47.441817 16615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a894243f25a04abb94c6111989f4de56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.441900 16615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:47.442051 16659 mvcc.cc:205] Tried to move back new op lower bound from 7333122733820866560 to 7333122733806678016. Current Snapshot: MvccSnapshot[applied={T|T < 7333122733820866560}]
I20260925 05:12:47.442523 16660 mvcc.cc:205] Tried to move back new op lower bound from 7333122733820866560 to 7333122733806678016. Current Snapshot: MvccSnapshot[applied={T|T < 7333122733820866560}]
I20260925 05:12:47.443099 16658 catalog_manager.cc:1347] Generated new cluster ID: 82e3d98b36a541c9800eb424f1b1d5cc
I20260925 05:12:47.443152 16658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:47.444077 16629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a894243f25a04abb94c6111989f4de56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.444149 16629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:12:47.443710 16643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a894243f25a04abb94c6111989f4de56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a894243f25a04abb94c6111989f4de56" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33531 } } peers { permanent_uuid: "88b4ef49da0842438f1176ecac3c3b8e" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 38863 } } peers { permanent_uuid: "f08a986ce38748ce87340299a3ff95eb" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 46497 } } }
I20260925 05:12:47.444344 16643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:47.444933 16662 mvcc.cc:205] Tried to move back new op lower bound from 7333122733820866560 to 7333122733806678016. Current Snapshot: MvccSnapshot[applied={T|T < 7333122733820866560}]
I20260925 05:12:47.451574 16658 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:47.452327 16658 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:47.458916 16658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a894243f25a04abb94c6111989f4de56: Generated new TSK 0
I20260925 05:12:47.459185 16658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:48.286420 16626 catalog_manager.cc:1259] Loaded cluster ID: 82e3d98b36a541c9800eb424f1b1d5cc
I20260925 05:12:48.286510 16626 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: loading cluster ID for follower catalog manager: success
I20260925 05:12:48.287297 16626 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: acquiring CA information for follower catalog manager: success
I20260925 05:12:48.287557 16626 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f08a986ce38748ce87340299a3ff95eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:12:48.309374 16640 catalog_manager.cc:1259] Loaded cluster ID: 82e3d98b36a541c9800eb424f1b1d5cc
I20260925 05:12:48.309459 16640 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: loading cluster ID for follower catalog manager: success
I20260925 05:12:48.310209 16640 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: acquiring CA information for follower catalog manager: success
I20260925 05:12:48.310467 16640 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88b4ef49da0842438f1176ecac3c3b8e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:12:48.514865 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16384
I20260925 05:12:48.520584 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16459
I20260925 05:12:48.526044 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16534
2026-09-25T05:12:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2966 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:48 dist-test-slave-1lvv krb5kdc[16674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:48 dist-test-slave-1lvv krb5kdc[16674](info): set up 2 sockets
Sep 25 05:12:48 dist-test-slave-1lvv krb5kdc[16674](info): commencing operation
krb5kdc: starting...
W20260925 05:12:49.564057 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:12:49 dist-test-slave-1lvv krb5kdc[16674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:12:49.695717 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:39405
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:39405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:49.779738 16687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:49.780107 16687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:49.780150 16687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:49.781391 16687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:49.781471 16687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:49.781498 16687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:49.781531 16687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:49.781556 16687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:49.783107 16687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:39405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:39405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:49.783404 16687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:49.783650 16687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:49.788776 16692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:49.788978 16687 server_base.cc:1034] running on GCE node
W20260925 05:12:49.788810 16693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:49.788780 16695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:49.789309 16687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:49.789631 16687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:49.790782 16687 hybrid_clock.cc:648] HybridClock initialized: now 1790313169790769 us; error 33 us; skew 500 ppm
Sep 25 05:12:49 dist-test-slave-1lvv krb5kdc[16674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313169, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:49.793402 16687 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:12:49.793936 16687 webserver.cc:460] Webserver started at http://127.3.6.190:39149/ using document root <none> and password file <none>
I20260925 05:12:49.794174 16687 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:49.794247 16687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:49.794418 16687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:49.795562 16687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "1d457db991ce47968fdfde6dd0f2af9b"
format_stamp: "Formatted at 2026-09-25 05:12:49 on dist-test-slave-1lvv"
server_key: "80469280dbecacf9954ff9f556e941d5"
server_key_iv: "017440a256900a3b92bac22830a27e48"
server_key_version: "encryptionkey@0"
I20260925 05:12:49.796013 16687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "1d457db991ce47968fdfde6dd0f2af9b"
format_stamp: "Formatted at 2026-09-25 05:12:49 on dist-test-slave-1lvv"
server_key: "80469280dbecacf9954ff9f556e941d5"
server_key_iv: "017440a256900a3b92bac22830a27e48"
server_key_version: "encryptionkey@0"
I20260925 05:12:49.797658 16687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:12:49.798506 16702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:49.798688 16687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:49.798811 16687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "1d457db991ce47968fdfde6dd0f2af9b"
format_stamp: "Formatted at 2026-09-25 05:12:49 on dist-test-slave-1lvv"
server_key: "80469280dbecacf9954ff9f556e941d5"
server_key_iv: "017440a256900a3b92bac22830a27e48"
server_key_version: "encryptionkey@0"
I20260925 05:12:49.798897 16687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:49.821214 16687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:49.823792 16687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:49.823982 16687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:49.828212 16687 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:39405
I20260925 05:12:49.828255 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:39405 every 8 connection(s)
I20260925 05:12:49.828641 16687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:49.829591 16755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:49.831436 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16687
I20260925 05:12:49.831558 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:12:49.831748 3098 external_mini_cluster.cc:1428] Setting key aa6cb8aaf1c686d3bf65d3df7cc36bff
I20260925 05:12:49.832497 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b: Bootstrap starting.
I20260925 05:12:49.833158 16755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:49.833498 16755 log.cc:824] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:49.834271 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b: No bootstrap required, opened a new log
I20260925 05:12:49.836683 16755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d457db991ce47968fdfde6dd0f2af9b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39405 } }
I20260925 05:12:49.836902 16755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:49.836971 16755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d457db991ce47968fdfde6dd0f2af9b, State: Initialized, Role: FOLLOWER
I20260925 05:12:49.837131 16755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d457db991ce47968fdfde6dd0f2af9b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39405 } }
I20260925 05:12:49.837198 16755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:49.837244 16755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:49.837292 16755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:49.837867 16755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d457db991ce47968fdfde6dd0f2af9b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39405 } }
I20260925 05:12:49.837997 16755 leader_election.cc:302] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d457db991ce47968fdfde6dd0f2af9b; no voters:
I20260925 05:12:49.838205 16755 leader_election.cc:288] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:49.838232 16760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:49.838357 16760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [term 1 LEADER]: Becoming Leader. State: Replica: 1d457db991ce47968fdfde6dd0f2af9b, State: Running, Role: LEADER
I20260925 05:12:49.838507 16755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:49.838521 16760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d457db991ce47968fdfde6dd0f2af9b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39405 } }
I20260925 05:12:49.839001 16762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d457db991ce47968fdfde6dd0f2af9b. Latest consensus state: current_term: 1 leader_uuid: "1d457db991ce47968fdfde6dd0f2af9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d457db991ce47968fdfde6dd0f2af9b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39405 } } }
I20260925 05:12:49.839098 16762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [sys.catalog]: This master's current role is: LEADER
Sep 25 05:12:49 dist-test-slave-1lvv krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:49.839322 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d457db991ce47968fdfde6dd0f2af9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d457db991ce47968fdfde6dd0f2af9b" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 39405 } } }
I20260925 05:12:49.839668 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:49.839932 16767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:49.840485 16767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:49.842698 16767 catalog_manager.cc:1347] Generated new cluster ID: 06fb3d89f599499db9921c385fea30c3
I20260925 05:12:49.842761 16767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:49.843927 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:49.832360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.3.6.190:39405)
0925 05:12:49.832546 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:49.832554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:49.832610 (+ 56us) server_negotiation.cc:407] Connection header received
0925 05:12:49.833088 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:49.833094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:49.833145 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:49.833320 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:49.834125 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.835442 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.836346 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:49.836816 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:49.836946 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:49.839882 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:49.839886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:49.839887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:49.839898 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:49.842147 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.842531 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.842535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.842535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.842659 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:49.842884 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:49.842886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:49.842886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:49.843211 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:49.843358 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:49.843496 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:12:49.843645 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260925 05:12:49.845158 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:12:49.859450 16767 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:49.860105 16767 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:49.864933 16767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d457db991ce47968fdfde6dd0f2af9b: Generated new TSK 0
I20260925 05:12:49.865152 16767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:12:49.899461 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16687
2026-09-25T05:12:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1371 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:12:49 dist-test-slave-1lvv krb5kdc[16782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:12:49 dist-test-slave-1lvv krb5kdc[16782](info): set up 2 sockets
Sep 25 05:12:49 dist-test-slave-1lvv krb5kdc[16782](info): commencing operation
krb5kdc: starting...
W20260925 05:12:50.932789 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:12:51.083264 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:34325
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:51.167945 16795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:51.168241 16795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:51.168286 16795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:51.170209 16795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:51.170284 16795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:51.170298 16795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:51.170313 16795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:51.170326 16795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:51.172034 16795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:34325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:51.172351 16795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:51.172617 16795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:51.177971 16800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:51.178263 16795 server_base.cc:1034] running on GCE node
W20260925 05:12:51.177985 16801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:51.177968 16803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:51.178597 16795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:51.178957 16795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:51.180116 16795 hybrid_clock.cc:648] HybridClock initialized: now 1790313171180097 us; error 39 us; skew 500 ppm
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:51.182762 16795 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:12:51.183275 16795 webserver.cc:460] Webserver started at http://127.3.6.190:36273/ using document root <none> and password file <none>
I20260925 05:12:51.183530 16795 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:51.183604 16795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:51.183813 16795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:51.184983 16795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "a6c1f0df4ebd4d3492beb77168ab05f1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "7c89bae992d37a94ff8e6b927b93242c"
server_key_iv: "6189611663496b5b4a335bb7fbf74b4b"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.185375 16795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "a6c1f0df4ebd4d3492beb77168ab05f1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "7c89bae992d37a94ff8e6b927b93242c"
server_key_iv: "6189611663496b5b4a335bb7fbf74b4b"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.186836 16795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:12:51.187629 16810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.187848 16795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:51.187956 16795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "a6c1f0df4ebd4d3492beb77168ab05f1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "7c89bae992d37a94ff8e6b927b93242c"
server_key_iv: "6189611663496b5b4a335bb7fbf74b4b"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.188043 16795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:51.195534 16795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:51.197589 16795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:51.197782 16795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:51.202119 16795 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:34325
I20260925 05:12:51.202154 16862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:34325 every 8 connection(s)
I20260925 05:12:51.202545 16795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:51.203615 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:51.206131 16863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Bootstrap starting.
I20260925 05:12:51.206845 16863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:51.207173 16863 log.cc:824] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:51.208138 16863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: No bootstrap required, opened a new log
I20260925 05:12:51.209035 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16795
I20260925 05:12:51.209168 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:12:51.209369 3098 external_mini_cluster.cc:1428] Setting key 56a390c3b8f950bed5a441b851b90e06
I20260925 05:12:51.210613 16863 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:51.210796 16863 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:51.210861 16863 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c1f0df4ebd4d3492beb77168ab05f1, State: Initialized, Role: FOLLOWER
I20260925 05:12:51.211035 16863 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:51.211117 16863 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:51.211141 16863 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:51.211213 16863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:51.212330 16863 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:51.212447 16863 leader_election.cc:302] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c1f0df4ebd4d3492beb77168ab05f1; no voters:
I20260925 05:12:51.212689 16863 leader_election.cc:288] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:12:51.212749 16868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:51.213006 16863 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:51.213016 16868 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 LEADER]: Becoming Leader. State: Replica: a6c1f0df4ebd4d3492beb77168ab05f1, State: Running, Role: LEADER
I20260925 05:12:51.213267 16868 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:51.213681 16870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } } }
I20260925 05:12:51.213805 16870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:51.214026 16870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c1f0df4ebd4d3492beb77168ab05f1. Latest consensus state: current_term: 1 leader_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } } }
I20260925 05:12:51.214093 16877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:51.214159 16870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:51.214720 16877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:51.216735 16877 catalog_manager.cc:1347] Generated new cluster ID: e2b2d2f52fcb45f081b94d5a287a74a7
I20260925 05:12:51.216801 16877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:51.222438 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.210003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.3.6.190:34325)
0925 05:12:51.210266 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.210275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.210332 (+ 57us) server_negotiation.cc:407] Connection header received
0925 05:12:51.210561 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.210567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.210616 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.210771 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:51.211598 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.212751 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.213976 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.214704 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.214829 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.218101 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:51.218106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:51.218106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:51.218120 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:51.220371 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.220821 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.220825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.220825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.220994 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.221211 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.221211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.221212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.221576 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:51.221767 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.221844 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.222013 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260925 05:12:51.233232 16877 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:12:51.234050 16877 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:51.239949 16877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Generated new TSK 0
I20260925 05:12:51.240197 16877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:12:51.271708 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:34325
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:51.357620 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:51.357863 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:51.357892 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:51.359103 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:51.359148 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:51.359175 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:12:51.360715 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:51.361007 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:51.361300 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:51.366685 16894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:51.366745 16895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:51.366935 16889 server_base.cc:1034] running on GCE node
W20260925 05:12:51.366688 16897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:51.367255 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:51.367497 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:51.368645 16889 hybrid_clock.cc:648] HybridClock initialized: now 1790313171368604 us; error 60 us; skew 500 ppm
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:51.371420 16889 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:12:51.372033 16889 webserver.cc:460] Webserver started at http://127.3.6.129:33859/ using document root <none> and password file <none>
I20260925 05:12:51.372251 16889 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:51.372298 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:51.372393 16889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:51.373543 16889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "0fe061132e32435eba9c82f0039d4028"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "16d4bc1367ccfe155ac3ec53b9a5b41b"
server_key_iv: "e16f46284d60a233d83b1d4077b225e8"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.373878 16889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "0fe061132e32435eba9c82f0039d4028"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "16d4bc1367ccfe155ac3ec53b9a5b41b"
server_key_iv: "e16f46284d60a233d83b1d4077b225e8"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.375352 16889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:12:51.376238 16904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.376423 16889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:12:51.376534 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "0fe061132e32435eba9c82f0039d4028"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "16d4bc1367ccfe155ac3ec53b9a5b41b"
server_key_iv: "e16f46284d60a233d83b1d4077b225e8"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.376614 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:51.391717 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:51.394052 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:51.394272 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:51.394539 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:51.394896 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:51.394963 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.395008 16889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:51.395033 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.401473 16889 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:41885
I20260925 05:12:51.401564 17017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:41885 every 8 connection(s)
I20260925 05:12:51.401918 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:12:51.407867 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 16889
I20260925 05:12:51.407959 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:12:51.408161 3098 external_mini_cluster.cc:1428] Setting key 3cfe96394de6d43f70e9c679938f9e31
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:51.412751 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.402775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:60623 (local address 127.3.6.190:34325)
0925 05:12:51.402845 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.402847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.403073 (+ 226us) server_negotiation.cc:407] Connection header received
0925 05:12:51.403633 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.403634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.403651 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.403711 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:51.404789 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.405320 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.406372 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.406592 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.406681 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.409295 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:51.409296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:51.409297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:51.409306 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:51.411055 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.411621 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.411621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.411622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.411677 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.412018 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.412019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.412020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.412133 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:51.412190 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.412569 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.412680 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:12:51.413045 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.402750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:60623)
0925 05:12:51.402937 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.402977 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.403515 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.403836 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.403843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.404462 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:51.404665 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.404671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.405428 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.405429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.406271 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.406274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.406695 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.406696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.406847 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.407428 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:51.407436 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:51.409191 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:51.411146 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:51.411155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:51.411159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:51.411532 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:51.411745 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:51.411746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:51.411746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:51.411943 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:51.412203 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:51.412210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:51.412416 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.412532 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.412741 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260925 05:12:51.413626 17018 heartbeater.cc:347] Connected to a master server at 127.3.6.190:34325
I20260925 05:12:51.413776 17018 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:51.414032 17018 heartbeater.cc:511] Master 127.3.6.190:34325 requested a full tablet report, sending...
I20260925 05:12:51.414639 16827 ts_manager.cc:194] Registered new tserver with Master: 0fe061132e32435eba9c82f0039d4028 (127.3.6.129:41885)
I20260925 05:12:51.415351 16827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:60623
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:12:51.439368 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:34325
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:51.524211 17023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:51.524518 17023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:51.524567 17023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:51.525774 17023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:51.525818 17023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:51.525846 17023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:12:51.527325 17023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:51.527549 17023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:51.527783 17023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:51.533159 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:51.533162 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:51.533162 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:51.533389 17023 server_base.cc:1034] running on GCE node
I20260925 05:12:51.533771 17023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:51.534091 17023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:51.535286 17023 hybrid_clock.cc:648] HybridClock initialized: now 1790313171535264 us; error 41 us; skew 500 ppm
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:51.538090 17023 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:12:51.538600 17023 webserver.cc:460] Webserver started at http://127.3.6.130:43921/ using document root <none> and password file <none>
I20260925 05:12:51.538869 17023 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:51.538945 17023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:51.539103 17023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:51.540333 17023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "b88b4555510340a5a4b1bf69aafbe8a1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "313484b510eb9579a0d5ccf116d42d76"
server_key_iv: "b104059c88d36234d019760afa94ab51"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.540722 17023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "b88b4555510340a5a4b1bf69aafbe8a1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "313484b510eb9579a0d5ccf116d42d76"
server_key_iv: "b104059c88d36234d019760afa94ab51"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.542160 17023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:12:51.542986 17038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.543164 17023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:12:51.543270 17023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "b88b4555510340a5a4b1bf69aafbe8a1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "313484b510eb9579a0d5ccf116d42d76"
server_key_iv: "b104059c88d36234d019760afa94ab51"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.543357 17023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:51.566144 17023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:51.568424 17023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:51.568617 17023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:51.568876 17023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:51.569221 17023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:51.569280 17023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.569324 17023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:51.569356 17023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.575610 17023 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:43135
I20260925 05:12:51.575656 17151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:43135 every 8 connection(s)
I20260925 05:12:51.576069 17023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:51.585579 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17023
I20260925 05:12:51.585743 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:12:51.585996 3098 external_mini_cluster.cc:1428] Setting key 1b1eae9f3ac1bf538affe6db3cfe075c
I20260925 05:12:51.586488 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.576970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:38509 (local address 127.3.6.190:34325)
0925 05:12:51.577073 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.577076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.577281 (+ 205us) server_negotiation.cc:407] Connection header received
0925 05:12:51.577854 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.577855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.577870 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.577928 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:51.578973 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.579571 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.580802 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.581015 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.581099 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.583477 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:51.583479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:51.583479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:51.583487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:51.584806 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.585238 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.585239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.585239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.585299 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.585671 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.585672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.585673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.585859 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:51.585914 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.586287 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.586384 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:12:51.586747 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.576969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:38509)
0925 05:12:51.577138 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.577185 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.577727 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.578001 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.578006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.578626 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:51.578849 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.578855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.579677 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.579678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.580703 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.580706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.581108 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.581110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.581263 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.581874 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:51.581883 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:51.583379 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:51.584903 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:51.584910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:51.584913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:51.585160 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:51.585409 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:51.585410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:51.585411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:51.585576 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:51.585915 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:51.585924 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:51.586120 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.586245 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.586464 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260925 05:12:51.587266 17152 heartbeater.cc:347] Connected to a master server at 127.3.6.190:34325
I20260925 05:12:51.587407 17152 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:51.587603 17152 heartbeater.cc:511] Master 127.3.6.190:34325 requested a full tablet report, sending...
I20260925 05:12:51.588070 16827 ts_manager.cc:194] Registered new tserver with Master: b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135)
I20260925 05:12:51.588474 16827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:38509
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:12:51.616223 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:34325
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:51.704550 17157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:51.704831 17157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:51.704877 17157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:51.706183 17157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:51.706269 17157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:51.706319 17157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:12:51.708170 17157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:51.708477 17157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:51.708750 17157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:51.714955 17162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:51.715030 17157 server_base.cc:1034] running on GCE node
W20260925 05:12:51.715148 17165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:51.715173 17163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:51.715484 17157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:51.715857 17157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:51.716979 17157 hybrid_clock.cc:648] HybridClock initialized: now 1790313171716969 us; error 33 us; skew 500 ppm
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:51.719750 17157 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:12:51.720296 17157 webserver.cc:460] Webserver started at http://127.3.6.131:41713/ using document root <none> and password file <none>
I20260925 05:12:51.720562 17157 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:51.720633 17157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:51.720798 17157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:12:51.721946 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "a742aace4e49453691b9e56f8ce09781"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "729e3a0e9dc6ca94fa68c8e7e9da7a8a"
server_key_iv: "dbb09fbcbd88a45d8a2be2a2d404f511"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.722344 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "a742aace4e49453691b9e56f8ce09781"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "729e3a0e9dc6ca94fa68c8e7e9da7a8a"
server_key_iv: "dbb09fbcbd88a45d8a2be2a2d404f511"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.723865 17157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:12:51.724691 17172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.724874 17157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:12:51.724978 17157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "a742aace4e49453691b9e56f8ce09781"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "729e3a0e9dc6ca94fa68c8e7e9da7a8a"
server_key_iv: "dbb09fbcbd88a45d8a2be2a2d404f511"
server_key_version: "encryptionkey@0"
I20260925 05:12:51.725066 17157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:51.747828 17157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:51.750319 17157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:51.750497 17157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:51.750764 17157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:51.751093 17157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:51.751150 17157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.751195 17157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:51.751221 17157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:51.757294 17157 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:37953
I20260925 05:12:51.757373 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:37953 every 8 connection(s)
I20260925 05:12:51.757715 17157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:12:51.762630 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17157
I20260925 05:12:51.762764 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:12:51.762984 3098 external_mini_cluster.cc:1428] Setting key 58b41024b7ece0bed042e2cdc3f050a0
Sep 25 05:12:51 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313171, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:51.767681 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.758517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:48609 (local address 127.3.6.190:34325)
0925 05:12:51.758594 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.758596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.758807 (+ 211us) server_negotiation.cc:407] Connection header received
0925 05:12:51.759319 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.759320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.759329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.759397 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:51.760520 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.761019 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.762078 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.762262 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.762319 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.765029 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:51.765031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:51.765031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:51.765039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:51.766373 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.766774 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.766775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.766775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.766833 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.767082 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.767083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.767083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.767229 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:51.767285 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.767544 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.767611 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:12:51.767906 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.758485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:48609)
0925 05:12:51.758667 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.758712 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.759229 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.759475 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.759479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.760175 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:51.760425 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.760430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.761110 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.761110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.761986 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.761988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.762369 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.762370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.762495 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.763164 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:51.763176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:51.764935 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:51.766455 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:51.766462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:51.766465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:51.766703 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:51.766915 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:51.766916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:51.766916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:51.767024 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:51.767289 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:51.767293 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:51.767425 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.767512 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.767642 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260925 05:12:51.768420 17286 heartbeater.cc:347] Connected to a master server at 127.3.6.190:34325
I20260925 05:12:51.768510 17286 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:51.768728 17286 heartbeater.cc:511] Master 127.3.6.190:34325 requested a full tablet report, sending...
I20260925 05:12:51.769140 16827 ts_manager.cc:194] Registered new tserver with Master: a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953)
I20260925 05:12:51.769529 16827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:48609
I20260925 05:12:51.774936 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:12:51.783111 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.776575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.3.6.190:34325)
0925 05:12:51.776673 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.776675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.776738 (+ 63us) server_negotiation.cc:407] Connection header received
0925 05:12:51.776787 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.776788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.776798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.776850 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:51.777429 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.777923 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.778712 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.778918 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.778977 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.780488 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:51.780490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:51.780490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:51.780498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:51.781950 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.782320 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.782321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.782322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.782380 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:51.782559 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:51.782560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:51.782560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:51.782715 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:51.782795 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.782908 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.782978 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:12:51.785051 16827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:12:51.786695 16827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:12:51.794226 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.789189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:43135 (local address 127.0.0.1:51342)
0925 05:12:51.789539 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.789603 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.789679 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.789974 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.789978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.789981 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:51.790076 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.790079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.791501 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.791501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.792845 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.792848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.793832 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.793833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.793931 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.793950 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.794020 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.794088 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":43,"thread_start_us":51,"threads_started":1}
I20260925 05:12:51.794483 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.789356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:41885 (local address 127.0.0.1:42582)
0925 05:12:51.789801 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.789813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.789900 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.790286 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.790287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.790289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:51.790326 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.790327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.791675 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.791676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.792794 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.792798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.793944 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.793946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.794016 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.794021 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.794101 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.794150 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260925 05:12:51.794480 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.789320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.3.6.130:43135)
0925 05:12:51.789774 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.789778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.789789 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:12:51.789840 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.789842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.789860 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.789953 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:51.790183 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.791398 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.792945 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.793718 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.794210 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.794297 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.794330 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.794383 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":108,"threads_started":1}
I20260925 05:12:51.795100 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.789532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.3.6.129:41885)
0925 05:12:51.789723 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.789729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.789869 (+ 140us) server_negotiation.cc:407] Connection header received
0925 05:12:51.790065 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.790068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.790084 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.790162 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:51.790413 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.791558 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.792920 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.793738 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.793793 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.793863 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.794153 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.794231 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260925 05:12:51.795321 17086 tablet_service.cc:1467] Processing CreateTablet for tablet e8f2566a8f164688854762b1a5dd0cb7 (DEFAULT_TABLE table=test-table [id=85fda30ed34b4e8299d9a5e17ebd7ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:51.795734 17086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f2566a8f164688854762b1a5dd0cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:51.795868 16952 tablet_service.cc:1467] Processing CreateTablet for tablet e8f2566a8f164688854762b1a5dd0cb7 (DEFAULT_TABLE table=test-table [id=85fda30ed34b4e8299d9a5e17ebd7ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:51.796278 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f2566a8f164688854762b1a5dd0cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:51.796742 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.789563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37953 (local address 127.0.0.1:35592)
0925 05:12:51.790064 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.790074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.790122 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.790338 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.790339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.790341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:51.790398 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.790400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.791367 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.791368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.793060 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.793062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.796493 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.796494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.796563 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.796569 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.796616 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.796669 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":34,"thread_start_us":63,"threads_started":1}
I20260925 05:12:51.796784 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.789684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.3.6.131:37953)
0925 05:12:51.789863 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.789867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.790188 (+ 321us) server_negotiation.cc:407] Connection header received
0925 05:12:51.790234 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.790237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.790253 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.790342 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:51.790458 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.791236 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.793152 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.796388 (+ 3236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.796491 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.796557 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.796644 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.796702 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260925 05:12:51.799444 17303 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Bootstrap starting.
I20260925 05:12:51.799952 17304 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Bootstrap starting.
I20260925 05:12:51.800069 17220 tablet_service.cc:1467] Processing CreateTablet for tablet e8f2566a8f164688854762b1a5dd0cb7 (DEFAULT_TABLE table=test-table [id=85fda30ed34b4e8299d9a5e17ebd7ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:51.800248 17303 tablet_bootstrap.cc:654] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:51.800407 17220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f2566a8f164688854762b1a5dd0cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:51.800787 17304 tablet_bootstrap.cc:654] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:51.800873 17303 log.cc:824] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:51.801308 17304 log.cc:824] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:51.802194 17304 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: No bootstrap required, opened a new log
I20260925 05:12:51.802264 17304 ts_tablet_manager.cc:1397] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:12:51.803231 17303 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: No bootstrap required, opened a new log
I20260925 05:12:51.803308 17303 ts_tablet_manager.cc:1397] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 05:12:51.803933 17307 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Bootstrap starting.
I20260925 05:12:51.804425 17304 raft_consensus.cc:348] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.804648 17304 raft_consensus.cc:374] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:51.804685 17307 tablet_bootstrap.cc:654] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:51.804699 17304 raft_consensus.cc:729] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe061132e32435eba9c82f0039d4028, State: Initialized, Role: FOLLOWER
I20260925 05:12:51.804816 17304 consensus_queue.cc:260] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.805122 17304 ts_tablet_manager.cc:1428] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:51.805123 17018 heartbeater.cc:503] Master 127.3.6.190:34325 was elected leader, sending a full tablet report...
I20260925 05:12:51.805294 17307 log.cc:824] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:51.805481 17303 raft_consensus.cc:348] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.805673 17303 raft_consensus.cc:374] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:51.805738 17303 raft_consensus.cc:729] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88b4555510340a5a4b1bf69aafbe8a1, State: Initialized, Role: FOLLOWER
I20260925 05:12:51.805891 17303 consensus_queue.cc:260] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.806206 17303 ts_tablet_manager.cc:1428] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:51.806344 17152 heartbeater.cc:503] Master 127.3.6.190:34325 was elected leader, sending a full tablet report...
I20260925 05:12:51.806628 17307 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: No bootstrap required, opened a new log
I20260925 05:12:51.806790 17307 ts_tablet_manager.cc:1397] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:12:51.808882 17307 raft_consensus.cc:348] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.809100 17307 raft_consensus.cc:374] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:51.809150 17307 raft_consensus.cc:729] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a742aace4e49453691b9e56f8ce09781, State: Initialized, Role: FOLLOWER
I20260925 05:12:51.809271 17307 consensus_queue.cc:260] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.809566 17307 ts_tablet_manager.cc:1428] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:12:51.809603 17286 heartbeater.cc:503] Master 127.3.6.190:34325 was elected leader, sending a full tablet report...
W20260925 05:12:51.826750 17153 tablet.cc:2369] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:12:51.902741 17019 tablet.cc:2369] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:51.967569 17308 raft_consensus.cc:482] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:51.967800 17308 raft_consensus.cc:504] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.968225 17308 leader_election.cc:288] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135), a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953)
I20260925 05:12:51.971623 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.968759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:37735 (local address 127.3.6.131:37953)
0925 05:12:51.968811 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.968813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.968820 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:12:51.968910 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.968911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.968925 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.968988 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:51.969203 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.969673 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.970862 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.971234 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.971327 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.971338 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.971493 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.971539 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:12:51.971623 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.968530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56187 (local address 127.3.6.130:43135)
0925 05:12:51.968605 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:51.968608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:51.968701 (+ 93us) server_negotiation.cc:407] Connection header received
0925 05:12:51.968818 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:51.968820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:51.968835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:51.968889 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:51.969111 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.969678 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.970818 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.971235 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.971327 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.971338 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:51.971504 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 05:12:51.971547 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:12:51.971658 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.968423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:43135 (local address 127.3.6.129:56187)
0925 05:12:51.968603 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.968618 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.968725 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.968928 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.968929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.968933 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:51.969027 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.969030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.969838 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.969838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.970717 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.970719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.971331 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.971332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.971413 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.971417 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.971483 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.971539 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":48,"thread_start_us":93,"threads_started":1}
I20260925 05:12:51.971668 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:51.968657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37953 (local address 127.3.6.129:37735)
0925 05:12:51.968747 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:51.968759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:51.968835 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:51.969009 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:51.969010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:51.969011 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:51.969116 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.969118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.969814 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.969816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.970736 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:51.970739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:51.971327 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:51.971328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:51.971405 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:51.971409 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:12:51.971486 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:12:51.971540 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":44,"thread_start_us":39,"threads_started":1}
I20260925 05:12:51.972132 17106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f2566a8f164688854762b1a5dd0cb7" candidate_uuid: "0fe061132e32435eba9c82f0039d4028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" is_pre_election: true
I20260925 05:12:51.972138 17240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f2566a8f164688854762b1a5dd0cb7" candidate_uuid: "0fe061132e32435eba9c82f0039d4028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a742aace4e49453691b9e56f8ce09781" is_pre_election: true
I20260925 05:12:51.972303 17106 raft_consensus.cc:2454] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe061132e32435eba9c82f0039d4028 in term 0.
I20260925 05:12:51.972303 17240 raft_consensus.cc:2454] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe061132e32435eba9c82f0039d4028 in term 0.
I20260925 05:12:51.972565 16907 leader_election.cc:302] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028, b88b4555510340a5a4b1bf69aafbe8a1; no voters:
I20260925 05:12:51.972806 17308 raft_consensus.cc:2781] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:12:51.972868 17308 raft_consensus.cc:482] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:12:51.972916 17308 raft_consensus.cc:3037] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:51.973677 17308 raft_consensus.cc:504] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.973827 17308 leader_election.cc:288] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 1 election: Requested vote from peers b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135), a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953)
I20260925 05:12:51.974047 17106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f2566a8f164688854762b1a5dd0cb7" candidate_uuid: "0fe061132e32435eba9c82f0039d4028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88b4555510340a5a4b1bf69aafbe8a1"
I20260925 05:12:51.974066 17240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f2566a8f164688854762b1a5dd0cb7" candidate_uuid: "0fe061132e32435eba9c82f0039d4028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a742aace4e49453691b9e56f8ce09781"
I20260925 05:12:51.974138 17106 raft_consensus.cc:3037] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:51.974162 17240 raft_consensus.cc:3037] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:51.975019 17106 raft_consensus.cc:2454] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe061132e32435eba9c82f0039d4028 in term 1.
I20260925 05:12:51.975044 17240 raft_consensus.cc:2454] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe061132e32435eba9c82f0039d4028 in term 1.
I20260925 05:12:51.975203 16907 leader_election.cc:302] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028, a742aace4e49453691b9e56f8ce09781; no voters:
I20260925 05:12:51.975322 17308 raft_consensus.cc:2781] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:51.975522 17308 raft_consensus.cc:686] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe061132e32435eba9c82f0039d4028, State: Running, Role: LEADER
I20260925 05:12:51.975656 17308 consensus_queue.cc:237] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:51.976675 16826 catalog_manager.cc:5495] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe061132e32435eba9c82f0039d4028 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "0fe061132e32435eba9c82f0039d4028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } health_report { overall_health: UNKNOWN } } }
W20260925 05:12:52.008387 17287 tablet.cc:2369] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:52.022667 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.020154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.3.6.129:41885)
0925 05:12:52.020264 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.020266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.020273 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:12:52.020312 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.020313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.020334 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.020417 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:52.020557 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.021046 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.021834 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.022043 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.022122 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.022309 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:52.022413 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:52.022517 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.022548 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.022570 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:12:52.024688 17106 raft_consensus.cc:1261] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 FOLLOWER]: Refusing update from remote peer 0fe061132e32435eba9c82f0039d4028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:52.024688 17240 raft_consensus.cc:1261] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Refusing update from remote peer 0fe061132e32435eba9c82f0039d4028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:52.025069 17308 consensus_queue.cc:1035] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:52.025266 17314 consensus_queue.cc:1035] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:52.028012 17318 mvcc.cc:205] Tried to move back new op lower bound from 7333122752610074624 to 7333122752413044736. Current Snapshot: MvccSnapshot[applied={T|T < 7333122752610074624}]
I20260925 05:12:52.031410 16827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:52.031560 16827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:52.032550 16827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Sending DeleteTablet for 3 replicas of tablet e8f2566a8f164688854762b1a5dd0cb7
I20260925 05:12:52.032950 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16889
I20260925 05:12:52.032975 16952 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f2566a8f164688854762b1a5dd0cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:52 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:42582
I20260925 05:12:52.033526 17220 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f2566a8f164688854762b1a5dd0cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:52 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:35592
I20260925 05:12:52.033831 17327 tablet_replica.cc:330] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: stopping tablet replica
I20260925 05:12:52.033967 17327 raft_consensus.cc:2229] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:52.034118 17327 raft_consensus.cc:2258] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:52.034291 17086 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f2566a8f164688854762b1a5dd0cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:52 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:51342
I20260925 05:12:52.034546 17328 tablet_replica.cc:330] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: stopping tablet replica
I20260925 05:12:52.034652 17327 ts_tablet_manager.cc:1905] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:52.034801 17328 raft_consensus.cc:2229] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:52.034979 17328 raft_consensus.cc:2258] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:52.036223 17328 ts_tablet_manager.cc:1905] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:52.036834 17327 ts_tablet_manager.cc:1918] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:52.036916 17327 log.cc:1196] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/e8f2566a8f164688854762b1a5dd0cb7
I20260925 05:12:52.037200 17327 ts_tablet_manager.cc:1939] T e8f2566a8f164688854762b1a5dd0cb7 P a742aace4e49453691b9e56f8ce09781: Deleting consensus metadata
I20260925 05:12:52.037730 16813 catalog_manager.cc:4841] TS a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953): tablet e8f2566a8f164688854762b1a5dd0cb7 (table test-table [id=85fda30ed34b4e8299d9a5e17ebd7ed7]) successfully deleted
I20260925 05:12:52.037771 17328 ts_tablet_manager.cc:1918] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:52.037859 17328 log.cc:1196] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/e8f2566a8f164688854762b1a5dd0cb7
I20260925 05:12:52.038128 17328 ts_tablet_manager.cc:1939] T e8f2566a8f164688854762b1a5dd0cb7 P b88b4555510340a5a4b1bf69aafbe8a1: Deleting consensus metadata
I20260925 05:12:52.038552 16813 catalog_manager.cc:4841] TS b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135): tablet e8f2566a8f164688854762b1a5dd0cb7 (table test-table [id=85fda30ed34b4e8299d9a5e17ebd7ed7]) successfully deleted
W20260925 05:12:52.040175 16813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:41885 (error 108)
I20260925 05:12:52.040350 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 17023
I20260925 05:12:52.041162 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.041032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:41885 (local address 127.0.0.1:42590)
0925 05:12:52.041091 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.041131 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 05:12:52.041337 16813 catalog_manager.cc:4584] TS 0fe061132e32435eba9c82f0039d4028 (127.3.6.129:41885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8f2566a8f164688854762b1a5dd0cb7: Network error: Client connection negotiation failed: client connection to 127.3.6.129:41885: connect: Connection refused (error 111)
I20260925 05:12:52.046983 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 17157
I20260925 05:12:52.052208 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 16795
I20260925 05:12:52.058054 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:34325
--webserver_interface=127.3.6.190
--webserver_port=36273
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:52.146289 17330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:52.146579 17330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:52.146624 17330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:52.147969 17330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:52.148048 17330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:52.148073 17330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:52.148094 17330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:52.148126 17330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:52.149804 17330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:34325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=36273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:52.150135 17330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:52.150395 17330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:52.160286 17337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:52.160204 17336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:52.160204 17339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:52.160645 17330 server_base.cc:1034] running on GCE node
I20260925 05:12:52.161051 17330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:52.161377 17330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:52.162544 17330 hybrid_clock.cc:648] HybridClock initialized: now 1790313172162519 us; error 36 us; skew 500 ppm
Sep 25 05:12:52 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:52.169845 17330 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:12:52.170470 17330 webserver.cc:460] Webserver started at http://127.3.6.190:36273/ using document root <none> and password file <none>
I20260925 05:12:52.170761 17330 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:52.170859 17330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:52.172544 17330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:12:52.173295 17346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.173491 17330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:12:52.173614 17330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "a6c1f0df4ebd4d3492beb77168ab05f1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "7c89bae992d37a94ff8e6b927b93242c"
server_key_iv: "6189611663496b5b4a335bb7fbf74b4b"
server_key_version: "encryptionkey@0"
I20260925 05:12:52.174093 17330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:52.225654 17330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:52.228067 17330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:52.228260 17330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:52.232439 17330 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:34325
I20260925 05:12:52.232513 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:34325 every 8 connection(s)
I20260925 05:12:52.232950 17330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:52.234725 17399 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:12:52.235044 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17330
I20260925 05:12:52.235388 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:41885
--local_ip_for_outbound_sockets=127.3.6.129
--tserver_master_addrs=127.3.6.190:34325
--webserver_port=33859
--webserver_interface=127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:52.235822 17399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Bootstrap starting.
I20260925 05:12:52.245282 17399 log.cc:824] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:52.247926 17399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:12:52.248361 17399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Bootstrap complete.
I20260925 05:12:52.250970 17399 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:52.251247 17399 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c1f0df4ebd4d3492beb77168ab05f1, State: Initialized, Role: FOLLOWER
I20260925 05:12:52.251439 17399 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:52.251520 17399 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:52.251593 17399 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:52.251677 17399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:12:52.252951 17399 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:52.253058 17399 leader_election.cc:302] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c1f0df4ebd4d3492beb77168ab05f1; no voters:
I20260925 05:12:52.253326 17399 leader_election.cc:288] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:12:52.253384 17403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:12:52.253575 17403 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 LEADER]: Becoming Leader. State: Replica: a6c1f0df4ebd4d3492beb77168ab05f1, State: Running, Role: LEADER
I20260925 05:12:52.253700 17399 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:52.253811 17403 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:52.254150 17405 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } } }
I20260925 05:12:52.254277 17405 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:52.254612 17411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:52.254146 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c1f0df4ebd4d3492beb77168ab05f1. Latest consensus state: current_term: 2 leader_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } } }
I20260925 05:12:52.254846 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:52.255714 17411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:52.255916 17411 catalog_manager.cc:1259] Loaded cluster ID: e2b2d2f52fcb45f081b94d5a287a74a7
I20260925 05:12:52.256153 17411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:12:52.256448 17411 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 05:12:52.260903 17411 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 05:12:52.263652 17411 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:12:52.263935 17411 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Loaded TSK: 0
I20260925 05:12:52.264166 17411 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:12:52.324909 17401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:52.325141 17401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:52.325168 17401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:52.326443 17401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:52.326499 17401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:52.326529 17401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:12:52.328176 17401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:41885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=33859
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:52.328460 17401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:52.328740 17401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:52.335063 17429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:52.335315 17401 server_base.cc:1034] running on GCE node
W20260925 05:12:52.335063 17426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:52.335088 17427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:52.335647 17401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:52.335947 17401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:52.337141 17401 hybrid_clock.cc:648] HybridClock initialized: now 1790313172337124 us; error 38 us; skew 500 ppm
Sep 25 05:12:52 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:52.339974 17401 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:12:52.340489 17401 webserver.cc:460] Webserver started at http://127.3.6.129:33859/ using document root <none> and password file <none>
I20260925 05:12:52.340754 17401 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:52.340894 17401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:52.342409 17401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:12:52.343158 17436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.343351 17401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:52.343468 17401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "0fe061132e32435eba9c82f0039d4028"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "16d4bc1367ccfe155ac3ec53b9a5b41b"
server_key_iv: "e16f46284d60a233d83b1d4077b225e8"
server_key_version: "encryptionkey@0"
I20260925 05:12:52.343997 17401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:52.364173 17401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:52.366839 17401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:52.367061 17401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:52.367334 17401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:52.367928 17443 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:12:52.370055 17401 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:12:52.370132 17401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 05:12:52.370180 17401 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:12:52.370877 17401 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:12:52.370944 17401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 05:12:52.371031 17443 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Bootstrap starting.
I20260925 05:12:52.378875 17401 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:41885
I20260925 05:12:52.379046 17550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:41885 every 8 connection(s)
I20260925 05:12:52.379385 17401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:12:52.381845 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17401
I20260925 05:12:52.382207 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:43135
--local_ip_for_outbound_sockets=127.3.6.130
--tserver_master_addrs=127.3.6.190:34325
--webserver_port=43921
--webserver_interface=127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:52.382962 17443 log.cc:824] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:52.387681 17443 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Sep 25 05:12:52 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:52.389060 17443 tablet_bootstrap.cc:492] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Bootstrap complete.
I20260925 05:12:52.391153 17443 ts_tablet_manager.cc:1397] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20260925 05:12:52.392797 17443 raft_consensus.cc:348] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:52.393271 17443 raft_consensus.cc:729] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe061132e32435eba9c82f0039d4028, State: Initialized, Role: FOLLOWER
I20260925 05:12:52.393510 17443 consensus_queue.cc:260] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } } peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } }
I20260925 05:12:52.393913 17443 ts_tablet_manager.cc:1428] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 05:12:52.394723 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.380395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:38293 (local address 127.3.6.190:34325)
0925 05:12:52.381313 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.381321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.381351 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:12:52.381469 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.381473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.381516 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.381663 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:52.382735 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.384244 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.385494 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.386174 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.386266 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.389650 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:52.389656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:52.389657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:52.389674 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:52.392223 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.392854 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.392859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.392860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.393019 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.393362 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.393363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.393363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.393712 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:52.393783 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.394277 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.394451 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":69,"threads_started":1}
I20260925 05:12:52.395394 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.380320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:38293)
0925 05:12:52.380636 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.380671 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.381144 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.381850 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.381854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.382278 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:52.382579 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.382587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.384382 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.384384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.385352 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.385356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.386417 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.386418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.386585 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.387287 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:52.387298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:52.389474 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:52.392350 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:52.392358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:52.392361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:52.392733 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:52.393116 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:52.393116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:52.393117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:52.393272 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:52.393846 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:52.393853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:52.394050 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.394927 (+ 877us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.395143 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260925 05:12:52.396260 17551 heartbeater.cc:347] Connected to a master server at 127.3.6.190:34325
I20260925 05:12:52.396375 17551 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:52.396596 17551 heartbeater.cc:511] Master 127.3.6.190:34325 requested a full tablet report, sending...
I20260925 05:12:52.397461 17363 ts_manager.cc:194] Registered new tserver with Master: 0fe061132e32435eba9c82f0039d4028 (127.3.6.129:41885)
I20260925 05:12:52.398743 17363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:38293
I20260925 05:12:52.399843 17551 heartbeater.cc:503] Master 127.3.6.190:34325 was elected leader, sending a full tablet report...
I20260925 05:12:52.404394 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.398192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:41885 (local address 127.0.0.1:42596)
0925 05:12:52.399101 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.399138 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.399223 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.399894 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.399898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.399900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:52.400007 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.400009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.401516 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.401517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.402889 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.402893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.404020 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.404022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.404136 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.404154 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.404238 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.404298 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":116,"threads_started":1}
I20260925 05:12:52.404404 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.398473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.3.6.129:41885)
0925 05:12:52.399380 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.399384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.399395 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:12:52.399434 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.399435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.399686 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.399811 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:52.400100 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.401407 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.403018 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.403919 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.404015 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.404081 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.404275 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.404326 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":71,"threads_started":1}
I20260925 05:12:52.405180 17485 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f2566a8f164688854762b1a5dd0cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:52 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:42596
I20260925 05:12:52.405872 17560 tablet_replica.cc:330] stopping tablet replica
I20260925 05:12:52.406000 17560 raft_consensus.cc:2229] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:52.406076 17560 raft_consensus.cc:2258] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:52.406454 17560 ts_tablet_manager.cc:1905] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:52.409355 17560 ts_tablet_manager.cc:1918] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:52.409463 17560 log.cc:1196] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/e8f2566a8f164688854762b1a5dd0cb7
I20260925 05:12:52.409824 17560 ts_tablet_manager.cc:1939] T e8f2566a8f164688854762b1a5dd0cb7 P 0fe061132e32435eba9c82f0039d4028: Deleting consensus metadata
I20260925 05:12:52.410509 17349 catalog_manager.cc:4841] TS 0fe061132e32435eba9c82f0039d4028 (127.3.6.129:41885): tablet e8f2566a8f164688854762b1a5dd0cb7 (table test-table [id=85fda30ed34b4e8299d9a5e17ebd7ed7]) successfully deleted
W20260925 05:12:52.476924 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:52.477187 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:52.477231 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:52.478600 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:52.478684 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:52.478734 17556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:12:52.480515 17556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:43135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=43921
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:52.480832 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:52.481110 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:52.486570 17565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:52.486613 17566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:52.486797 17556 server_base.cc:1034] running on GCE node
W20260925 05:12:52.486583 17568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:52.487114 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:52.487449 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:52.488620 17556 hybrid_clock.cc:648] HybridClock initialized: now 1790313172488596 us; error 39 us; skew 500 ppm
Sep 25 05:12:52 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:52.491266 17556 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:12:52.491873 17556 webserver.cc:460] Webserver started at http://127.3.6.130:43921/ using document root <none> and password file <none>
I20260925 05:12:52.492149 17556 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:52.492244 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:52.493738 17556 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:12:52.494441 17575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.494648 17556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:12:52.494761 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "b88b4555510340a5a4b1bf69aafbe8a1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "313484b510eb9579a0d5ccf116d42d76"
server_key_iv: "b104059c88d36234d019760afa94ab51"
server_key_version: "encryptionkey@0"
I20260925 05:12:52.495157 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:52.525800 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:52.528136 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:52.528321 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:52.528595 17556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:52.528934 17556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:52.528993 17556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.529038 17556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:52.529091 17556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.535612 17556 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:43135
I20260925 05:12:52.535665 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:43135 every 8 connection(s)
I20260925 05:12:52.536101 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:12:52.540493 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17556
I20260925 05:12:52.540808 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:37953
--local_ip_for_outbound_sockets=127.3.6.131
--tserver_master_addrs=127.3.6.190:34325
--webserver_port=41713
--webserver_interface=127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:12:52 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:52.546535 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.536956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:41623 (local address 127.3.6.190:34325)
0925 05:12:52.537010 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.537012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.537281 (+ 269us) server_negotiation.cc:407] Connection header received
0925 05:12:52.537771 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.537772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.537785 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.537841 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:52.538846 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.539348 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.540452 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.540787 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.540879 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.543282 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:52.543284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:52.543284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:52.543292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:52.544957 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.545419 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.545420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.545421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.545491 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.545787 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.545788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.545789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.545921 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:52.546002 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.546329 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.546434 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:12:52.546756 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.536957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:41623)
0925 05:12:52.537139 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.537179 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.537675 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.537917 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.537920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.538545 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:52.538753 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.538757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.539434 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.539434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.540355 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.540358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.540902 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.540903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.541082 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.541706 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:52.541715 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:52.543146 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:52.545045 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:52.545056 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:52.545059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:52.545331 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:52.545564 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:52.545565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:52.545566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:52.545701 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:52.545992 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:52.546000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:52.546171 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.546307 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.546493 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260925 05:12:52.547282 17689 heartbeater.cc:347] Connected to a master server at 127.3.6.190:34325
I20260925 05:12:52.547386 17689 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:52.547605 17689 heartbeater.cc:511] Master 127.3.6.190:34325 requested a full tablet report, sending...
I20260925 05:12:52.548126 17363 ts_manager.cc:194] Registered new tserver with Master: b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135)
I20260925 05:12:52.548739 17363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:41623
W20260925 05:12:52.628089 17692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:52.628325 17692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:52.628352 17692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:52.629557 17692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:52.629612 17692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:52.629640 17692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:12:52.631283 17692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:37953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=41713
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:52.631510 17692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:52.631711 17692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:52.637203 17700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:52.637207 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:52.637243 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:52.637512 17692 server_base.cc:1034] running on GCE node
I20260925 05:12:52.637791 17692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:52.638087 17692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:52.639318 17692 hybrid_clock.cc:648] HybridClock initialized: now 1790313172639314 us; error 47 us; skew 500 ppm
Sep 25 05:12:52 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:52.642102 17692 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:12:52.642623 17692 webserver.cc:460] Webserver started at http://127.3.6.131:41713/ using document root <none> and password file <none>
I20260925 05:12:52.642879 17692 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:52.642980 17692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:52.644516 17692 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:52.645172 17707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.645356 17692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:12:52.645470 17692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "a742aace4e49453691b9e56f8ce09781"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "729e3a0e9dc6ca94fa68c8e7e9da7a8a"
server_key_iv: "dbb09fbcbd88a45d8a2be2a2d404f511"
server_key_version: "encryptionkey@0"
I20260925 05:12:52.645864 17692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:52.672966 17692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:52.675711 17692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:52.675944 17692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:52.676247 17692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:52.676586 17692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:52.676642 17692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.676677 17692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:52.676699 17692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.683193 17692 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:37953
I20260925 05:12:52.683228 17820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:37953 every 8 connection(s)
I20260925 05:12:52.683638 17692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:12:52.687225 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17692
Sep 25 05:12:52 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:12:52.695021 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.684512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:47569 (local address 127.3.6.190:34325)
0925 05:12:52.684587 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.684589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.684817 (+ 228us) server_negotiation.cc:407] Connection header received
0925 05:12:52.685418 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.685420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.685432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.685495 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:52.686488 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.687025 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.688356 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.688573 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.688667 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.691174 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:52.691175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:52.691176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:52.691184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:52.693139 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.693589 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.693591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.693591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.693652 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.694082 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.694083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.694084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.694262 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:52.694345 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.694729 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.694899 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":46}
I20260925 05:12:52.695188 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.684498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:47569)
0925 05:12:52.684685 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.684724 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.685301 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.685574 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.685577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.686177 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:52.686383 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.686388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.687135 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.687136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.688259 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.688262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.688677 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.688679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.688811 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.689388 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:52.689398 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:52.691072 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:52.693234 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:52.693241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:52.693243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:52.693512 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:52.693824 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:52.693825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:52.693826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:52.693998 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:52.694361 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:52.694370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:52.694564 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.694706 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.694902 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260925 05:12:52.695084 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.687744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.3.6.190:34325)
0925 05:12:52.687903 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.687905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.688033 (+ 128us) server_negotiation.cc:407] Connection header received
0925 05:12:52.688119 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.688120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.688133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.688220 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:52.688842 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.689335 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.690244 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.690446 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.690514 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.692083 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:52.692084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:52.692084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:52.692092 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:52.693537 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.694082 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.694083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.694083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.694182 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:52.694398 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:52.694399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:52.694399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:52.694507 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:52.694624 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.694728 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.694960 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":133,"thread_start_us":109,"threads_started":1}
I20260925 05:12:52.696009 17821 heartbeater.cc:347] Connected to a master server at 127.3.6.190:34325
I20260925 05:12:52.696146 17821 heartbeater.cc:464] Registering TS with master...
I20260925 05:12:52.696348 17821 heartbeater.cc:511] Master 127.3.6.190:34325 requested a full tablet report, sending...
I20260925 05:12:52.696735 17363 ts_manager.cc:194] Registered new tserver with Master: a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953)
I20260925 05:12:52.697080 17363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:47569
I20260925 05:12:52.707304 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:12:52.710170 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.707722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.3.6.190:34325)
0925 05:12:52.707823 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.707826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.707950 (+ 124us) server_negotiation.cc:407] Connection header received
0925 05:12:52.708010 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.708011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.708050 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.708120 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:52.708235 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.708727 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.709494 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.709696 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.709753 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.709889 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:52.709996 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:52.710056 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.710086 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.710106 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:12:52.710494 17363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:12:52.712234 17363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:12:52.714959 17485 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7fe7887754cda929d0bd6df8dbcd3 (DEFAULT_TABLE table=test-table [id=24bb16a9229f45ce9526902398f6a6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:52.715209 17485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7fe7887754cda929d0bd6df8dbcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:52.716617 17443 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Bootstrap starting.
I20260925 05:12:52.716922 17443 tablet_bootstrap.cc:654] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:52.717723 17443 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: No bootstrap required, opened a new log
I20260925 05:12:52.717794 17443 ts_tablet_manager.cc:1397] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:12:52.717933 17443 raft_consensus.cc:348] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.717988 17443 raft_consensus.cc:374] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:52.718021 17443 raft_consensus.cc:729] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe061132e32435eba9c82f0039d4028, State: Initialized, Role: FOLLOWER
I20260925 05:12:52.718082 17443 consensus_queue.cc:260] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.718180 17443 ts_tablet_manager.cc:1428] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:52.718648 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.714395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37953 (local address 127.0.0.1:35598)
0925 05:12:52.714475 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.714489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.714541 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.714885 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.714886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.714887 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:52.714942 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.714944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.716274 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.716275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.717430 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.717432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.718370 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.718372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.718466 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.718471 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.718528 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.718577 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:12:52.718921 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.714503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.3.6.131:37953)
0925 05:12:52.714704 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.714709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.714718 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:12:52.714772 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.714774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.714787 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.714887 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:52.715011 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.716154 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.717533 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.718278 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.718368 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.718431 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.718781 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.718834 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":129,"threads_started":1}
I20260925 05:12:52.718998 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.714508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:43135 (local address 127.0.0.1:51354)
0925 05:12:52.714630 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.714641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.714690 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.715285 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.715286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.715289 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:52.715424 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.715427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.716694 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.716695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.717732 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.717734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.718717 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.718718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.718794 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.718797 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.718854 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.718894 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260925 05:12:52.719244 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.714842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.3.6.130:43135)
0925 05:12:52.715044 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.715049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.715060 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:12:52.715100 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.715103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.715118 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.715199 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:52.715498 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.716582 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.717835 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.718545 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.718611 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.718666 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.719098 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.719150 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
I20260925 05:12:52.719674 17755 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7fe7887754cda929d0bd6df8dbcd3 (DEFAULT_TABLE table=test-table [id=24bb16a9229f45ce9526902398f6a6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:52.719913 17623 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7fe7887754cda929d0bd6df8dbcd3 (DEFAULT_TABLE table=test-table [id=24bb16a9229f45ce9526902398f6a6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:12:52.720096 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7fe7887754cda929d0bd6df8dbcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:52.720271 17623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7fe7887754cda929d0bd6df8dbcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:12:52.723052 17831 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Bootstrap starting.
I20260925 05:12:52.723419 17832 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Bootstrap starting.
I20260925 05:12:52.723842 17831 tablet_bootstrap.cc:654] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:52.724093 17832 tablet_bootstrap.cc:654] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Neither blocks nor log segments found. Creating new log.
I20260925 05:12:52.724435 17831 log.cc:824] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:52.724630 17832 log.cc:824] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:52.725250 17831 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: No bootstrap required, opened a new log
I20260925 05:12:52.725309 17831 ts_tablet_manager.cc:1397] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:12:52.725440 17832 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: No bootstrap required, opened a new log
I20260925 05:12:52.725632 17832 ts_tablet_manager.cc:1397] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:12:52.726964 17831 raft_consensus.cc:348] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.727128 17831 raft_consensus.cc:374] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:52.727152 17831 raft_consensus.cc:729] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a742aace4e49453691b9e56f8ce09781, State: Initialized, Role: FOLLOWER
I20260925 05:12:52.727072 17832 raft_consensus.cc:348] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.727288 17832 raft_consensus.cc:374] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:12:52.727285 17831 consensus_queue.cc:260] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.727351 17832 raft_consensus.cc:729] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88b4555510340a5a4b1bf69aafbe8a1, State: Initialized, Role: FOLLOWER
I20260925 05:12:52.727473 17832 consensus_queue.cc:260] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.727610 17821 heartbeater.cc:503] Master 127.3.6.190:34325 was elected leader, sending a full tablet report...
I20260925 05:12:52.727612 17831 ts_tablet_manager.cc:1428] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:12:52.727910 17832 ts_tablet_manager.cc:1428] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:12:52.727923 17689 heartbeater.cc:503] Master 127.3.6.190:34325 was elected leader, sending a full tablet report...
W20260925 05:12:52.786736 17690 tablet.cc:2369] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:52.856045 17836 raft_consensus.cc:482] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:52.856217 17836 raft_consensus.cc:504] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.856609 17836 leader_election.cc:288] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953), 0fe061132e32435eba9c82f0039d4028 (127.3.6.129:41885)
I20260925 05:12:52.859745 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.857082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:47025 (local address 127.3.6.129:41885)
0925 05:12:52.857131 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.857133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.857253 (+ 120us) server_negotiation.cc:407] Connection header received
0925 05:12:52.857291 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.857293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.857304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.857363 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:52.857531 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.858036 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.859008 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.859364 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.859447 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.859458 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.859607 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.859655 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:12:52.859783 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.856813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37953 (local address 127.3.6.130:47865)
0925 05:12:52.856898 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.856912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.856961 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.857157 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.857158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.857161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:52.857288 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.857290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.858030 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.858031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.858866 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.858868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.859479 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.859480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.859555 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.859558 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.859635 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.859673 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":42}
I20260925 05:12:52.859783 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.856914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:47865 (local address 127.3.6.131:37953)
0925 05:12:52.856988 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.856990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.857004 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:12:52.857037 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.857038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.857050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.857149 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:12:52.857378 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.857937 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.858970 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.859367 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.859457 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.859467 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.859656 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.859698 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:12:52.859793 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.856987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:41885 (local address 127.3.6.130:47025)
0925 05:12:52.857169 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.857181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:52.857239 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:52.857371 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:52.857372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:52.857373 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:12:52.857461 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.857464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.858120 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.858121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.858886 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:52.858888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.859449 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.859450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:52.859522 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.859525 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:12:52.859579 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:12:52.859632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260925 05:12:52.860141 17505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fe7887754cda929d0bd6df8dbcd3" candidate_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe061132e32435eba9c82f0039d4028" is_pre_election: true
I20260925 05:12:52.860230 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fe7887754cda929d0bd6df8dbcd3" candidate_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a742aace4e49453691b9e56f8ce09781" is_pre_election: true
I20260925 05:12:52.860361 17505 raft_consensus.cc:2454] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88b4555510340a5a4b1bf69aafbe8a1 in term 0.
I20260925 05:12:52.860409 17775 raft_consensus.cc:2454] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88b4555510340a5a4b1bf69aafbe8a1 in term 0.
I20260925 05:12:52.860620 17578 leader_election.cc:302] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028, b88b4555510340a5a4b1bf69aafbe8a1; no voters:
I20260925 05:12:52.860778 17836 raft_consensus.cc:2781] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:12:52.860862 17836 raft_consensus.cc:482] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:12:52.860882 17836 raft_consensus.cc:3037] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:52.861682 17836 raft_consensus.cc:504] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.861837 17836 leader_election.cc:288] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [CANDIDATE]: Term 1 election: Requested vote from peers a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953), 0fe061132e32435eba9c82f0039d4028 (127.3.6.129:41885)
I20260925 05:12:52.862071 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fe7887754cda929d0bd6df8dbcd3" candidate_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a742aace4e49453691b9e56f8ce09781"
I20260925 05:12:52.862071 17505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fe7887754cda929d0bd6df8dbcd3" candidate_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe061132e32435eba9c82f0039d4028"
I20260925 05:12:52.862174 17775 raft_consensus.cc:3037] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:52.862165 17505 raft_consensus.cc:3037] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:12:52.863190 17775 raft_consensus.cc:2454] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88b4555510340a5a4b1bf69aafbe8a1 in term 1.
I20260925 05:12:52.863375 17578 leader_election.cc:302] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a742aace4e49453691b9e56f8ce09781, b88b4555510340a5a4b1bf69aafbe8a1; no voters:
I20260925 05:12:52.863539 17836 raft_consensus.cc:2781] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:12:52.863658 17505 raft_consensus.cc:2454] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88b4555510340a5a4b1bf69aafbe8a1 in term 1.
I20260925 05:12:52.863719 17836 raft_consensus.cc:686] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 LEADER]: Becoming Leader. State: Replica: b88b4555510340a5a4b1bf69aafbe8a1, State: Running, Role: LEADER
I20260925 05:12:52.863867 17836 consensus_queue.cc:237] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:52.864743 17363 catalog_manager.cc:5495] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130). New cstate: current_term: 1 leader_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } health_report { overall_health: UNKNOWN } } }
W20260925 05:12:52.880216 17552 tablet.cc:2369] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:12:52.904152 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.901657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.3.6.130:43135)
0925 05:12:52.901712 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:52.901714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:52.901725 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:12:52.901815 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:52.901816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:52.901838 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:52.901917 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:12:52.902045 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.902535 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.903322 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:52.903517 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:52.903607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:52.903809 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:12:52.903913 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:12:52.904014 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:52.904034 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 05:12:52.904055 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:12:52.906036 17505 raft_consensus.cc:1261] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Refusing update from remote peer b88b4555510340a5a4b1bf69aafbe8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:52.906057 17775 raft_consensus.cc:1261] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Refusing update from remote peer b88b4555510340a5a4b1bf69aafbe8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:12:52.906412 17836 consensus_queue.cc:1035] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:52.906594 17838 consensus_queue.cc:1035] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:12:52.909505 17844 mvcc.cc:205] Tried to move back new op lower bound from 7333122756220239872 to 7333122756051070976. Current Snapshot: MvccSnapshot[applied={T|T < 7333122756220239872 or (T in {7333122756220239872})}]
I20260925 05:12:52.910871 17363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:52.910967 17363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:12:52.911773 17363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Sending DeleteTablet for 3 replicas of tablet e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:12:52.912120 17485 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c7fe7887754cda929d0bd6df8dbcd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:52 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:42596
I20260925 05:12:52.912199 17623 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c7fe7887754cda929d0bd6df8dbcd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:52 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:51354
I20260925 05:12:52.912250 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 17401
I20260925 05:12:52.912309 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c7fe7887754cda929d0bd6df8dbcd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:12:52 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:35598
I20260925 05:12:52.912982 17852 tablet_replica.cc:330] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: stopping tablet replica
I20260925 05:12:52.913103 17852 raft_consensus.cc:2229] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:12:52.913216 17852 raft_consensus.cc:2258] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:52.913379 17851 tablet_replica.cc:330] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: stopping tablet replica
I20260925 05:12:52.913602 17851 raft_consensus.cc:2229] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:12:52.913698 17852 ts_tablet_manager.cc:1905] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:52.913758 17851 raft_consensus.cc:2258] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:12:52.914286 17851 ts_tablet_manager.cc:1905] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:12:52.915845 17852 ts_tablet_manager.cc:1918] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:52.915933 17852 log.cc:1196] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:12:52.916268 17851 ts_tablet_manager.cc:1918] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:12:52.916290 17852 ts_tablet_manager.cc:1939] T e0c7fe7887754cda929d0bd6df8dbcd3 P a742aace4e49453691b9e56f8ce09781: Deleting consensus metadata
I20260925 05:12:52.916364 17851 log.cc:1196] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:12:52.916631 17851 ts_tablet_manager.cc:1939] T e0c7fe7887754cda929d0bd6df8dbcd3 P b88b4555510340a5a4b1bf69aafbe8a1: Deleting consensus metadata
I20260925 05:12:52.916819 17349 catalog_manager.cc:4841] TS a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953): tablet e0c7fe7887754cda929d0bd6df8dbcd3 (table test-table [id=24bb16a9229f45ce9526902398f6a6d4]) successfully deleted
I20260925 05:12:52.917039 17349 catalog_manager.cc:4841] TS b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135): tablet e0c7fe7887754cda929d0bd6df8dbcd3 (table test-table [id=24bb16a9229f45ce9526902398f6a6d4]) successfully deleted
W20260925 05:12:52.920439 17349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:41885 (error 108)
I20260925 05:12:52.920791 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 17556
I20260925 05:12:52.921701 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:52.921557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:41885 (local address 127.0.0.1:42600)
0925 05:12:52.921609 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:52.921657 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 05:12:52.921846 17349 catalog_manager.cc:4584] TS 0fe061132e32435eba9c82f0039d4028 (127.3.6.129:41885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0c7fe7887754cda929d0bd6df8dbcd3: Network error: Client connection negotiation failed: client connection to 127.3.6.129:41885: connect: Connection refused (error 111)
I20260925 05:12:52.926445 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 17692
I20260925 05:12:52.931962 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 17330
I20260925 05:12:52.937881 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:34325
--webserver_interface=127.3.6.190
--webserver_port=36273
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:53.024773 17854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:53.024996 17854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:53.025020 17854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:53.026211 17854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:12:53.026262 17854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:53.026278 17854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:53.026321 17854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:12:53.026351 17854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:12:53.027904 17854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:34325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=36273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:53.028191 17854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:53.028451 17854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:53.033866 17860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:53.033839 17862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:53.033854 17859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:53.034152 17854 server_base.cc:1034] running on GCE node
I20260925 05:12:53.034466 17854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:53.034793 17854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:53.035975 17854 hybrid_clock.cc:648] HybridClock initialized: now 1790313173035963 us; error 35 us; skew 500 ppm
Sep 25 05:12:53 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:53.038702 17854 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:12:53.039150 17854 webserver.cc:460] Webserver started at http://127.3.6.190:36273/ using document root <none> and password file <none>
I20260925 05:12:53.039351 17854 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:53.039403 17854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:53.040900 17854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:53.041842 17869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.042047 17854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:53.042136 17854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "a6c1f0df4ebd4d3492beb77168ab05f1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "7c89bae992d37a94ff8e6b927b93242c"
server_key_iv: "6189611663496b5b4a335bb7fbf74b4b"
server_key_version: "encryptionkey@0"
I20260925 05:12:53.042516 17854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:53.057335 17854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:53.059981 17854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:53.060117 17854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:53.064328 17854 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:34325
I20260925 05:12:53.064415 17921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:34325 every 8 connection(s)
I20260925 05:12:53.064823 17854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:12:53.066541 17922 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:12:53.067198 17922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Bootstrap starting.
I20260925 05:12:53.073952 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17854
I20260925 05:12:53.074273 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:41885
--local_ip_for_outbound_sockets=127.3.6.129
--tserver_master_addrs=127.3.6.190:34325
--webserver_port=33859
--webserver_interface=127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:53.076813 17922 log.cc:824] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:53.079201 17922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:12:53.079553 17922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1: Bootstrap complete.
I20260925 05:12:53.081964 17922 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:53.082249 17922 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c1f0df4ebd4d3492beb77168ab05f1, State: Initialized, Role: FOLLOWER
I20260925 05:12:53.082465 17922 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:53.082549 17922 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:12:53.082608 17922 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:12:53.082662 17922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:12:53.083729 17922 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:53.083894 17922 leader_election.cc:302] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c1f0df4ebd4d3492beb77168ab05f1; no voters:
I20260925 05:12:53.084131 17922 leader_election.cc:288] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:12:53.084172 17926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:12:53.084293 17926 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [term 3 LEADER]: Becoming Leader. State: Replica: a6c1f0df4ebd4d3492beb77168ab05f1, State: Running, Role: LEADER
I20260925 05:12:53.084420 17922 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:12:53.084470 17926 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } }
I20260925 05:12:53.085168 17927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } } }
I20260925 05:12:53.085260 17927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:53.085732 17928 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c1f0df4ebd4d3492beb77168ab05f1. Latest consensus state: current_term: 3 leader_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c1f0df4ebd4d3492beb77168ab05f1" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 34325 } } }
I20260925 05:12:53.085999 17935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:12:53.085821 17928 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c1f0df4ebd4d3492beb77168ab05f1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:12:53.086870 17935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:12:53.086988 17935 catalog_manager.cc:1259] Loaded cluster ID: e2b2d2f52fcb45f081b94d5a287a74a7
I20260925 05:12:53.087057 17935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 05:12:53.087378 17935 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc147bb29ad google::LogMessage::Fail() at ??:0
@ 0x7fc147bb588a google::LogMessage::SendToLog() at ??:0
@ 0x7fc147bb24de google::LogMessage::Flush() at ??:0
@ 0x7fc147bb3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc14fad30f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc14fadc8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc149a800d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc149a75954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc14a8d96db start_thread at ??:0
@ 0x7fc1466b061f clone at ??:0
W20260925 05:12:53.178812 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:53.179057 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:53.179086 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:53.180440 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:53.180495 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:53.180526 17924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:12:53.182159 17924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:41885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=33859
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:53.182449 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:53.182723 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:53.188035 17959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:53.188062 17960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:53.188225 17962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:53.188762 17924 server_base.cc:1034] running on GCE node
I20260925 05:12:53.188979 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:53.189246 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:53.190438 17924 hybrid_clock.cc:648] HybridClock initialized: now 1790313173190428 us; error 42 us; skew 500 ppm
Sep 25 05:12:53 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:53.193634 17924 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:12:53.194144 17924 webserver.cc:460] Webserver started at http://127.3.6.129:33859/ using document root <none> and password file <none>
I20260925 05:12:53.194407 17924 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:53.194483 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:53.196002 17924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:12:53.196774 17969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.196955 17924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:53.197079 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "0fe061132e32435eba9c82f0039d4028"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "16d4bc1367ccfe155ac3ec53b9a5b41b"
server_key_iv: "e16f46284d60a233d83b1d4077b225e8"
server_key_version: "encryptionkey@0"
I20260925 05:12:53.197460 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:53.209933 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:53.212534 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:53.212736 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:53.213006 17924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:53.213521 17976 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:12:53.215744 17924 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:12:53.215916 17924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260925 05:12:53.215974 17924 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:12:53.216775 17924 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:12:53.216845 17924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 05:12:53.216919 17976 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Bootstrap starting.
I20260925 05:12:53.225962 17924 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:41885
I20260925 05:12:53.226059 18083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:41885 every 8 connection(s)
I20260925 05:12:53.226456 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:12:53.229568 17976 log.cc:824] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Log is configured to *not* fsync() on all Append() calls
I20260925 05:12:53.230875 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 17924
I20260925 05:12:53.231166 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:43135
--local_ip_for_outbound_sockets=127.3.6.130
--tserver_master_addrs=127.3.6.190:34325
--webserver_port=43921
--webserver_interface=127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:12:53.232842 17976 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:12:53.233179 17976 tablet_bootstrap.cc:492] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Bootstrap complete.
I20260925 05:12:53.234619 17976 ts_tablet_manager.cc:1397] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.010s
I20260925 05:12:53.236395 17976 raft_consensus.cc:348] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:53.236752 17976 raft_consensus.cc:729] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe061132e32435eba9c82f0039d4028, State: Initialized, Role: FOLLOWER
I20260925 05:12:53.237001 17976 consensus_queue.cc:260] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:53.237336 17976 ts_tablet_manager.cc:1428] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 05:12:53.317221 18088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:53.317512 18088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:53.317554 18088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:53.318742 18088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:53.318806 18088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:53.318850 18088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:12:53.320391 18088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:43135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=43921
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:53.320650 18088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:53.320890 18088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:53.326081 18094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:53.326061 18095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:53.326512 18088 server_base.cc:1034] running on GCE node
W20260925 05:12:53.326056 18097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:53.326889 18088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:53.327230 18088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:53.328433 18088 hybrid_clock.cc:648] HybridClock initialized: now 1790313173328406 us; error 50 us; skew 500 ppm
Sep 25 05:12:53 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:53.331740 18088 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:12:53.332399 18088 webserver.cc:460] Webserver started at http://127.3.6.130:43921/ using document root <none> and password file <none>
I20260925 05:12:53.332670 18088 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:53.332741 18088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:53.334591 18088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:12:53.335480 18104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.335711 18088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:53.335839 18088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "b88b4555510340a5a4b1bf69aafbe8a1"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "313484b510eb9579a0d5ccf116d42d76"
server_key_iv: "b104059c88d36234d019760afa94ab51"
server_key_version: "encryptionkey@0"
I20260925 05:12:53.336310 18088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260925 05:12:53.345690 18086 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:12:53.227333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:44069)
0925 05:12:53.227555 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.227597 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:53.228261 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:53.345457 (+117196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":122,"threads_started":1}
W20260925 05:12:53.345994 18084 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:34325 (0 consecutive failures): Network error: Failed to ping master at 127.3.6.190:34325: Client connection negotiation failed: client connection to 127.3.6.190:34325: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:12:53.346354 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.346235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:48271)
0925 05:12:53.346291 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.346335 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:12:53.346688 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.346601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:59263)
0925 05:12:53.346647 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.346672 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:12:53.346817 18084 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:12:53.358868 18088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:53.360942 18088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:53.361132 18088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:53.361409 18088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:53.361745 18088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:53.361802 18088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.361850 18088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:53.361876 18088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.368220 18088 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:43135
I20260925 05:12:53.368264 18217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:43135 every 8 connection(s)
I20260925 05:12:53.368669 18088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:12:53.369894 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.369491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:53301)
0925 05:12:53.369657 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.369734 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20260925 05:12:53.370343 18218 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:34325 (0 consecutive failures): Network error: Failed to ping master at 127.3.6.190:34325: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
I20260925 05:12:53.370644 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.370551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:35359)
0925 05:12:53.370598 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.370626 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 05:12:53.370939 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.370871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:53107)
0925 05:12:53.370908 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.370927 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 05:12:53.371086 18218 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:12:53.378445 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 18088
I20260925 05:12:53.378809 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:37953
--local_ip_for_outbound_sockets=127.3.6.131
--tserver_master_addrs=127.3.6.190:34325
--webserver_port=41713
--webserver_interface=127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:12:53.464641 18221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:12:53.464890 18221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:12:53.464917 18221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:12:53.466213 18221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:12:53.466274 18221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:12:53.466305 18221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:12:53.468003 18221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:37953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=41713
--tserver_master_addrs=127.3.6.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:12:53.468283 18221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:12:53.468538 18221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:12:53.473903 18226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:12:53.473903 18227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:53.474083 18221 server_base.cc:1034] running on GCE node
W20260925 05:12:53.473973 18229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:12:53.474445 18221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:12:53.474726 18221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:12:53.475945 18221 hybrid_clock.cc:648] HybridClock initialized: now 1790313173475932 us; error 45 us; skew 500 ppm
Sep 25 05:12:53 dist-test-slave-1lvv krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:12:53.478578 18221 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:12:53.479099 18221 webserver.cc:460] Webserver started at http://127.3.6.131:41713/ using document root <none> and password file <none>
I20260925 05:12:53.479328 18221 fs_manager.cc:359] Metadata directory not provided
I20260925 05:12:53.479378 18221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:12:53.480862 18221 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:53.481530 18236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.481740 18221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:12:53.481845 18221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "a742aace4e49453691b9e56f8ce09781"
format_stamp: "Formatted at 2026-09-25 05:12:51 on dist-test-slave-1lvv"
server_key: "729e3a0e9dc6ca94fa68c8e7e9da7a8a"
server_key_iv: "dbb09fbcbd88a45d8a2be2a2d404f511"
server_key_version: "encryptionkey@0"
I20260925 05:12:53.482213 18221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:12:53.494902 18221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:12:53.497148 18221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:12:53.497285 18221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:12:53.497509 18221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:12:53.497877 18221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:12:53.497924 18221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.497972 18221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:12:53.497999 18221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:12:53.504470 18221 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:37953
I20260925 05:12:53.504562 18349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:37953 every 8 connection(s)
I20260925 05:12:53.504957 18221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:12:53.506224 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.505784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:42575)
0925 05:12:53.505958 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.506041 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
W20260925 05:12:53.506680 18350 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:34325 (0 consecutive failures): Network error: Failed to ping master at 127.3.6.190:34325: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
I20260925 05:12:53.507004 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.506893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:32833)
0925 05:12:53.506954 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.506985 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:12:53.507246 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:53.507196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:59407)
0925 05:12:53.507217 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:53.507235 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 05:12:53.507362 18350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:12:53.515166 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 18221
I20260925 05:12:54.347577 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:54.347288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:55879)
0925 05:12:54.347458 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:54.347510 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260925 05:12:54.371740 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:54.371497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:49235)
0925 05:12:54.371636 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:54.371680 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260925 05:12:54.508019 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:54.507737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:49893)
0925 05:12:54.507928 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:54.507968 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":125,"threads_started":1}
I20260925 05:12:54.770469 18357 raft_consensus.cc:482] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:54.770653 18357 raft_consensus.cc:504] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:54.771018 18357 leader_election.cc:288] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953), b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135)
Sep 25 05:12:54 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.131@KRBTEST.COM
Sep 25 05:12:54 dist-test-slave-1lvv krb5kdc[16782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313173, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.130@KRBTEST.COM
I20260925 05:12:54.782569 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:54.771518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:45239 (local address 127.3.6.130:43135)
0925 05:12:54.771665 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:54.771675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:54.771715 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:12:54.772205 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:54.772207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:54.772244 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:54.772411 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:54.773149 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.774001 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.775014 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.775577 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.775679 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:54.778636 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:54.778640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:54.778640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:54.778652 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:54.780579 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:54.781033 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:54.781037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:54.781037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:54.781169 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:54.781509 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:54.781510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:54.781510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:54.781854 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:54.781946 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:54.782309 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 05:12:54.782451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":50,"threads_started":1}
I20260925 05:12:54.782569 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:54.771379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:60875 (local address 127.3.6.131:37953)
0925 05:12:54.771577 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:12:54.771588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:12:54.771618 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:12:54.772158 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:12:54.772160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:12:54.772213 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:12:54.772362 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:12:54.773140 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.773993 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.775000 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.775575 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.775705 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:54.778262 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:12:54.778284 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:12:54.778285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:12:54.778299 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:12:54.780401 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:54.781034 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:54.781037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:54.781037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:54.781170 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:12:54.781509 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:12:54.781510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:12:54.781510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:12:54.781854 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:12:54.781946 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:12:54.782309 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 05:12:54.782451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260925 05:12:54.782641 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:54.771253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37953 (local address 127.3.6.129:60875)
0925 05:12:54.771334 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:54.771352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:54.771438 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:54.772417 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:54.772420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:54.772828 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:54.773015 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:54.773022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.774098 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.774099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:54.774865 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:54.774875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.775671 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.775672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:54.775813 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:54.776518 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:54.776529 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:54.778137 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:54.780485 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:54.780493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:54.780495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:54.780921 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:54.781260 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:54.781261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:54.781261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:54.781431 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:54.781935 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:54.781942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:54.782111 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 05:12:54.782234 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 05:12:54.782460 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":19}
I20260925 05:12:54.782650 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:54.771428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:43135 (local address 127.3.6.129:45239)
0925 05:12:54.771591 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:54.771602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:12:54.771657 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:12:54.772430 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:12:54.772431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:12:54.772827 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:12:54.773015 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:54.773022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.774126 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.774126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:54.774873 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:12:54.774876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:12:54.775802 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:12:54.775803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:12:54.775856 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:12:54.776570 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:12:54.776574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:12:54.778522 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:12:54.780656 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:54.780657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:54.780657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:54.780953 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:54.781264 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:12:54.781264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:12:54.781265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:12:54.781431 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:12:54.781935 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:12:54.781942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:12:54.782111 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 05:12:54.782234 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 05:12:54.782520 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":193,"thread_start_us":93,"threads_started":1}
W20260925 05:12:54.783510 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
W20260925 05:12:54.784219 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:12:54.784267 17972 leader_election.cc:302] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028; no voters: a742aace4e49453691b9e56f8ce09781, b88b4555510340a5a4b1bf69aafbe8a1
I20260925 05:12:54.784406 18357 raft_consensus.cc:2726] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:12:55.348353 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:55.348141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:49467)
0925 05:12:55.348266 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:55.348305 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260925 05:12:55.372498 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:55.372283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:40285)
0925 05:12:55.372410 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:55.372457 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":103,"threads_started":1}
I20260925 05:12:55.508642 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:55.508443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:39307)
0925 05:12:55.508568 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:55.508606 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 05:12:56.349109 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:56.348864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:40263)
0925 05:12:56.349011 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:56.349056 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260925 05:12:56.373309 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:56.373043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:46387)
0925 05:12:56.373207 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:56.373246 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260925 05:12:56.397744 18366 raft_consensus.cc:482] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:56.397851 18366 raft_consensus.cc:504] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:56.398061 18366 leader_election.cc:288] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953), b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135)
W20260925 05:12:56.398589 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
W20260925 05:12:56.398705 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:12:56.398792 17972 leader_election.cc:302] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028; no voters: a742aace4e49453691b9e56f8ce09781, b88b4555510340a5a4b1bf69aafbe8a1
I20260925 05:12:56.398923 18366 raft_consensus.cc:2726] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:12:56.509464 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:56.509223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:47093)
0925 05:12:56.509378 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:56.509421 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260925 05:12:57.349922 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:57.349682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:48689)
0925 05:12:57.349819 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:57.349874 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260925 05:12:57.374053 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:57.373834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:50841)
0925 05:12:57.373979 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:57.374011 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260925 05:12:57.510227 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:57.510017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:57287)
0925 05:12:57.510150 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:57.510186 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260925 05:12:58.350701 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:58.350459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:43525)
0925 05:12:58.350612 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:58.350653 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 05:12:58.374902 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:58.374633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:34245)
0925 05:12:58.374818 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:58.374861 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":134,"threads_started":1}
I20260925 05:12:58.510982 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:58.510746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:57961)
0925 05:12:58.510887 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:58.510932 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260925 05:12:58.876616 18374 raft_consensus.cc:482] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:12:58.876730 18374 raft_consensus.cc:504] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:12:58.876926 18374 leader_election.cc:288] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953), b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135)
W20260925 05:12:58.877480 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
W20260925 05:12:58.877599 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:12:58.877626 17972 leader_election.cc:302] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028; no voters: a742aace4e49453691b9e56f8ce09781, b88b4555510340a5a4b1bf69aafbe8a1
I20260925 05:12:58.877720 18374 raft_consensus.cc:2726] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:12:59.351512 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:59.351250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:43309)
0925 05:12:59.351412 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:59.351460 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260925 05:12:59.375932 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:59.375589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:53959)
0925 05:12:59.375797 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:59.375856 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260925 05:12:59.511875 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:12:59.511561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:47727)
0925 05:12:59.511707 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:12:59.511826 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260925 05:13:00.352516 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:00.352225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:53113)
0925 05:13:00.352410 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:00.352460 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260925 05:13:00.376669 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:00.376444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:50837)
0925 05:13:00.376576 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:00.376621 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260925 05:13:00.512713 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:00.512461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:47235)
0925 05:13:00.512620 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:00.512667 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260925 05:13:01.353302 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:01.353024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:41735)
0925 05:13:01.353203 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:01.353249 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260925 05:13:01.377419 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:01.377151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:40551)
0925 05:13:01.377330 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:01.377371 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":119,"threads_started":1}
I20260925 05:13:01.513511 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:01.513267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:46327)
0925 05:13:01.513419 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:01.513463 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260925 05:13:01.853505 18384 raft_consensus.cc:482] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:13:01.853610 18384 raft_consensus.cc:504] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:13:01.853742 18384 leader_election.cc:288] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953), b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135)
W20260925 05:13:01.854310 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
W20260925 05:13:01.854420 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:13:01.854440 17972 leader_election.cc:302] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028; no voters: a742aace4e49453691b9e56f8ce09781, b88b4555510340a5a4b1bf69aafbe8a1
I20260925 05:13:01.854519 18384 raft_consensus.cc:2726] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:13:02.354102 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:02.353840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:37609)
0925 05:13:02.354002 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:02.354050 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260925 05:13:02.378237 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:02.377976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:47405)
0925 05:13:02.378150 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:02.378189 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260925 05:13:02.514338 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:02.514073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:60439)
0925 05:13:02.514234 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:02.514286 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260925 05:13:03.354872 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:03.354620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:47983)
0925 05:13:03.354775 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:03.354821 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260925 05:13:03.379110 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:03.378827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:55989)
0925 05:13:03.379024 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:03.379064 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":126,"threads_started":1}
I20260925 05:13:03.515239 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:03.514977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:48861)
0925 05:13:03.515134 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:03.515182 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260925 05:13:04.355698 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:04.355418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:53481)
0925 05:13:04.355592 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:04.355647 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260925 05:13:04.379961 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:04.379690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:50645)
0925 05:13:04.379862 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:04.379908 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":226,"threads_started":1}
I20260925 05:13:04.516036 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:04.515740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:59113)
0925 05:13:04.515953 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:04.515990 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":151,"threads_started":1}
I20260925 05:13:05.356545 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:05.356322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:60325)
0925 05:13:05.356459 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:05.356496 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260925 05:13:05.380725 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:05.380481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:49327)
0925 05:13:05.380623 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:05.380677 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260925 05:13:05.516734 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:05.516501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:50045)
0925 05:13:05.516646 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:05.516686 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260925 05:13:06.357319 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:06.357029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:40303)
0925 05:13:06.357228 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:06.357272 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260925 05:13:06.381630 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:06.381283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:40231)
0925 05:13:06.381515 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:06.381570 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260925 05:13:06.517584 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:06.517322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:60331)
0925 05:13:06.517463 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:06.517514 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":53,"threads_started":1}
I20260925 05:13:07.358237 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:07.357973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:60067)
0925 05:13:07.358130 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:07.358185 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260925 05:13:07.382445 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:07.382180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:47583)
0925 05:13:07.382353 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:07.382399 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
I20260925 05:13:07.406677 18402 raft_consensus.cc:482] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:13:07.406785 18402 raft_consensus.cc:504] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a742aace4e49453691b9e56f8ce09781" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37953 } } peers { permanent_uuid: "b88b4555510340a5a4b1bf69aafbe8a1" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 43135 } } peers { permanent_uuid: "0fe061132e32435eba9c82f0039d4028" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 41885 } }
I20260925 05:13:07.407001 18402 leader_election.cc:288] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953), b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135)
W20260925 05:13:07.407419 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a742aace4e49453691b9e56f8ce09781 (127.3.6.131:37953): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
W20260925 05:13:07.407506 17972 leader_election.cc:341] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88b4555510340a5a4b1bf69aafbe8a1 (127.3.6.130:43135): Not found: Tablet not found: e0c7fe7887754cda929d0bd6df8dbcd3
I20260925 05:13:07.407523 17972 leader_election.cc:302] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe061132e32435eba9c82f0039d4028; no voters: a742aace4e49453691b9e56f8ce09781, b88b4555510340a5a4b1bf69aafbe8a1
I20260925 05:13:07.407599 18402 raft_consensus.cc:2726] T e0c7fe7887754cda929d0bd6df8dbcd3 P 0fe061132e32435eba9c82f0039d4028 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:13:07.518538 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:07.518263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:48955)
0925 05:13:07.518430 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:07.518486 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260925 05:13:08.359212 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:08.358956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.129:34063)
0925 05:13:08.359110 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:08.359159 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260925 05:13:08.383252 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:08.383033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.130:33153)
0925 05:13:08.383160 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:08.383205 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260925 05:13:08.519537 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:08.519288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:34325 (local address 127.3.6.131:48481)
0925 05:13:08.519446 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:08.519490 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260925 05:13:08.520912 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 17924
I20260925 05:13:08.526311 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 18088
I20260925 05:13:08.531459 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 18221
2026-09-25T05:13:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18636 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:13:08 dist-test-slave-1lvv krb5kdc[18409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:13:08 dist-test-slave-1lvv krb5kdc[18409](info): set up 2 sockets
Sep 25 05:13:08 dist-test-slave-1lvv krb5kdc[18409](info): commencing operation
krb5kdc: starting...
W20260925 05:13:09.576092 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:13:09 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:13:09.711969 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32979
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:32979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:13:09.796172 18422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:09.796466 18422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:09.796510 18422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:09.797791 18422 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:13:09.797869 18422 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:13:09.797891 18422 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:13:09.797912 18422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:13:09.797941 18422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:09.797976 18422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:09.798000 18422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:13:09.798031 18422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:13:09.799619 18422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:32979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:09.799961 18422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:09.800220 18422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:09.806236 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:09.806306 18428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:09.806365 18427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:09.806331 18422 server_base.cc:1034] running on GCE node
I20260925 05:13:09.806740 18422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:09.807097 18422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:09.808287 18422 hybrid_clock.cc:648] HybridClock initialized: now 1790313189808276 us; error 55 us; skew 500 ppm
Sep 25 05:13:09 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313189, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:09.811005 18422 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:13:09.811554 18422 webserver.cc:460] Webserver started at http://127.3.6.190:37407/ using document root <none> and password file <none>
I20260925 05:13:09.811856 18422 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:09.811931 18422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:09.812114 18422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:13:09.813285 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "9a8b78fa59d14e4586b2f6597e793c08"
format_stamp: "Formatted at 2026-09-25 05:13:09 on dist-test-slave-1lvv"
server_key: "6110eaec16ba8fcc59770bcfedc4860d"
server_key_iv: "a64c6891d2d5f82edc8161b8c6dad477"
server_key_version: "encryptionkey@0"
I20260925 05:13:09.813699 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "9a8b78fa59d14e4586b2f6597e793c08"
format_stamp: "Formatted at 2026-09-25 05:13:09 on dist-test-slave-1lvv"
server_key: "6110eaec16ba8fcc59770bcfedc4860d"
server_key_iv: "a64c6891d2d5f82edc8161b8c6dad477"
server_key_version: "encryptionkey@0"
I20260925 05:13:09.815212 18422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:13:09.816083 18437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:09.816300 18422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:09.816433 18422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "9a8b78fa59d14e4586b2f6597e793c08"
format_stamp: "Formatted at 2026-09-25 05:13:09 on dist-test-slave-1lvv"
server_key: "6110eaec16ba8fcc59770bcfedc4860d"
server_key_iv: "a64c6891d2d5f82edc8161b8c6dad477"
server_key_version: "encryptionkey@0"
I20260925 05:13:09.816514 18422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:09.832679 18422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:09.834765 18422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:09.834950 18422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:09.839246 18422 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:32979
I20260925 05:13:09.839289 18489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:32979 every 8 connection(s)
I20260925 05:13:09.839677 18422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:13:09.840641 18490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:13:09.843360 18490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Bootstrap starting.
I20260925 05:13:09.844084 18490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Neither blocks nor log segments found. Creating new log.
I20260925 05:13:09.844417 18490 log.cc:824] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:09.845355 18490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: No bootstrap required, opened a new log
I20260925 05:13:09.847054 18490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:09.847214 18490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:13:09.847240 18490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8b78fa59d14e4586b2f6597e793c08, State: Initialized, Role: FOLLOWER
I20260925 05:13:09.847414 18490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:09.847486 18490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:13:09.847528 18490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:13:09.847579 18490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:13:09.848120 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 18422
I20260925 05:13:09.848244 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:13:09.848455 3098 external_mini_cluster.cc:1428] Setting key 4b3ac0c63c90a5e6735d21e5c7eeac27
I20260925 05:13:09.848450 18490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:09.848556 18490 leader_election.cc:302] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a8b78fa59d14e4586b2f6597e793c08; no voters:
I20260925 05:13:09.848753 18490 leader_election.cc:288] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:13:09.848836 18493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:13:09.849014 18493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 LEADER]: Becoming Leader. State: Replica: 9a8b78fa59d14e4586b2f6597e793c08, State: Running, Role: LEADER
I20260925 05:13:09.849062 18490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:13:09.849205 18493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:09.849711 18493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a8b78fa59d14e4586b2f6597e793c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } } }
I20260925 05:13:09.849786 18493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: This master's current role is: LEADER
I20260925 05:13:09.849910 18493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8b78fa59d14e4586b2f6597e793c08. Latest consensus state: current_term: 1 leader_uuid: "9a8b78fa59d14e4586b2f6597e793c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } } }
I20260925 05:13:09.849965 18493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: This master's current role is: LEADER
I20260925 05:13:09.850338 18502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:13:09.851110 18502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:13:09.853682 18502 catalog_manager.cc:1347] Generated new cluster ID: 5236ea3a1e794dcd94d5d229895ad202
I20260925 05:13:09.853737 18502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:13:09 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:13:09.861302 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:09.849487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.3.6.190:32979)
0925 05:13:09.849658 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:09.849670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:09.849716 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:13:09.849857 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:09.849866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:09.849936 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:09.850719 (+ 783us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:09.851011 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:09.852405 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:09.853336 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:09.853957 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:09.854089 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:09.856938 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:09.856942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:09.856943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:09.856955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:09.859530 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:09.860001 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:09.860004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:09.860005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:09.860126 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:09.860351 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:09.860351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:09.860352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:09.860661 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:09.860799 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:09.860937 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:13:09.861079 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260925 05:13:09.873624 18502 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:13:09.874395 18502 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:13:09.879730 18502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Generated new TSK 0
I20260925 05:13:09.879990 18502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:13:09.915835 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:32979
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:13:10.000571 18516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:10.000805 18516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:10.000833 18516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:10.002024 18516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:10.002068 18516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:10.002096 18516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:13:10.003566 18516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:10.003875 18516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:10.004133 18516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:10.009405 18522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:10.009423 18521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:10.009450 18516 server_base.cc:1034] running on GCE node
W20260925 05:13:10.009631 18524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:10.009862 18516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:10.010149 18516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:10.011350 18516 hybrid_clock.cc:648] HybridClock initialized: now 1790313190011333 us; error 33 us; skew 500 ppm
Sep 25 05:13:10 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313190, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:10.014362 18516 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:13:10.014899 18516 webserver.cc:460] Webserver started at http://127.3.6.129:44953/ using document root <none> and password file <none>
I20260925 05:13:10.015168 18516 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:10.015240 18516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:10.015403 18516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:13:10.016589 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "9a054633e36c468d83130306818630a1"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "693a43691d90e6209a6a8f3537167505"
server_key_iv: "8ab3ca97c88c662db90b08f0ba87c88f"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.016974 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "9a054633e36c468d83130306818630a1"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "693a43691d90e6209a6a8f3537167505"
server_key_iv: "8ab3ca97c88c662db90b08f0ba87c88f"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.018472 18516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:13:10.019537 18531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.019783 18516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:13:10.019945 18516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "9a054633e36c468d83130306818630a1"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "693a43691d90e6209a6a8f3537167505"
server_key_iv: "8ab3ca97c88c662db90b08f0ba87c88f"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.020041 18516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:10.054584 18516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:10.056903 18516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:10.057109 18516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:10.057382 18516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:10.057739 18516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:13:10.057796 18516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.057832 18516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:13:10.057857 18516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.064546 18516 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:37449
I20260925 05:13:10.064587 18644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:37449 every 8 connection(s)
I20260925 05:13:10.065037 18516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:13:10 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313190, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:13:10.072659 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 18516
I20260925 05:13:10.072818 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:13:10.073083 3098 external_mini_cluster.cc:1428] Setting key 4310694337bacc0ab040a51f1d3c5f2f
I20260925 05:13:10.076548 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.065949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:42921 (local address 127.3.6.190:32979)
0925 05:13:10.066022 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.066024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.066279 (+ 255us) server_negotiation.cc:407] Connection header received
0925 05:13:10.066863 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.066864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.066880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.066950 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:10.068169 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.068728 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.069820 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.070020 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.070081 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.072743 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:10.072745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:10.072745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:10.072757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:10.074547 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.075187 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.075188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.075188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.075252 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.075618 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.075619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.075620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.075853 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:10.075953 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.076287 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.076419 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:13:10.076575 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.065940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:42921)
0925 05:13:10.066130 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.066178 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.066706 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.067142 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.067146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.067808 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:10.068024 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.068029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.068843 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.068843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.069725 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.069728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.070111 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.070139 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.070268 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.070821 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:10.070830 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:10.072603 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:10.074644 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:10.074655 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:10.074657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:10.075048 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:10.075337 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:10.075338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:10.075338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:10.075513 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:10.075954 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:10.075963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:10.076131 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.076236 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.076385 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260925 05:13:10.077227 18645 heartbeater.cc:347] Connected to a master server at 127.3.6.190:32979
I20260925 05:13:10.077361 18645 heartbeater.cc:464] Registering TS with master...
I20260925 05:13:10.077571 18645 heartbeater.cc:511] Master 127.3.6.190:32979 requested a full tablet report, sending...
I20260925 05:13:10.078238 18454 ts_manager.cc:194] Registered new tserver with Master: 9a054633e36c468d83130306818630a1 (127.3.6.129:37449)
I20260925 05:13:10.079275 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:42921
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:13:10.103751 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:32979
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:13:10.188838 18650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:10.189134 18650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:10.189193 18650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:10.190480 18650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:10.190572 18650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:10.190621 18650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:13:10.192301 18650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:10.192621 18650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:10.192888 18650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:10.198467 18656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:10.198452 18658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:10.198452 18655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:10.198637 18650 server_base.cc:1034] running on GCE node
I20260925 05:13:10.199043 18650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:10.199345 18650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:10.200523 18650 hybrid_clock.cc:648] HybridClock initialized: now 1790313190200513 us; error 43 us; skew 500 ppm
Sep 25 05:13:10 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313190, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:10.203255 18650 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:13:10.203835 18650 webserver.cc:460] Webserver started at http://127.3.6.130:45387/ using document root <none> and password file <none>
I20260925 05:13:10.204085 18650 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:10.204155 18650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:10.204314 18650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:13:10.205430 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "10a751c234ae46d2923344de5d229a5b"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a0b7c7739a20e135f487ba5e7738db68"
server_key_iv: "d3674918fedab1ac625d764391fa8c3e"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.205862 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "10a751c234ae46d2923344de5d229a5b"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a0b7c7739a20e135f487ba5e7738db68"
server_key_iv: "d3674918fedab1ac625d764391fa8c3e"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.207288 18650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:13:10.208176 18665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.208410 18650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:10.208526 18650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "10a751c234ae46d2923344de5d229a5b"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a0b7c7739a20e135f487ba5e7738db68"
server_key_iv: "d3674918fedab1ac625d764391fa8c3e"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.208609 18650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:10.230023 18650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:10.232709 18650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:10.232916 18650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:10.233184 18650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:10.233533 18650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:13:10.233594 18650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.233631 18650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:13:10.233680 18650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.239650 18650 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:46609
I20260925 05:13:10.239718 18778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:46609 every 8 connection(s)
I20260925 05:13:10.240074 18650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:13:10 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313190, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:13:10.249979 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 18650
I20260925 05:13:10.250080 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:13:10.250324 3098 external_mini_cluster.cc:1428] Setting key 8a9ded59b00acb1fdead90745d12f142
I20260925 05:13:10.250799 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.240916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:35639 (local address 127.3.6.190:32979)
0925 05:13:10.241025 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.241028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.241200 (+ 172us) server_negotiation.cc:407] Connection header received
0925 05:13:10.241777 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.241779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.241791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.241887 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:10.242893 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.243413 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.244557 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.244789 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.244884 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.247564 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:10.247566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:10.247567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:10.247583 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:10.248969 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.249522 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.249524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.249524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.249597 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.249947 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.249948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.249948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.250135 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:10.250243 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.250595 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.250703 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:13:10.251049 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.240831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:35639)
0925 05:13:10.241066 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.241106 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.241661 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.241976 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.241979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.242583 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:10.242784 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.242788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.243519 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.243520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.244450 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.244453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.244893 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.244894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.245032 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.245756 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:10.245766 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:10.247434 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:10.249072 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:10.249083 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:10.249086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:10.249417 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:10.249689 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:10.249689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:10.249690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:10.249857 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:10.250234 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:10.250243 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:10.250432 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.250572 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.250761 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260925 05:13:10.251655 18779 heartbeater.cc:347] Connected to a master server at 127.3.6.190:32979
I20260925 05:13:10.251792 18779 heartbeater.cc:464] Registering TS with master...
I20260925 05:13:10.252020 18779 heartbeater.cc:511] Master 127.3.6.190:32979 requested a full tablet report, sending...
I20260925 05:13:10.252446 18454 ts_manager.cc:194] Registered new tserver with Master: 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609)
I20260925 05:13:10.252837 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:35639
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:13:10.280519 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:32979
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:13:10.367285 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:10.367535 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:10.367563 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:10.368834 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:10.368884 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:10.368913 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:13:10.370427 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:10.370653 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:10.370865 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:10.376092 18790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:10.376374 18784 server_base.cc:1034] running on GCE node
W20260925 05:13:10.376094 18789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:10.376085 18792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:10.376684 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:10.376958 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:10.378168 18784 hybrid_clock.cc:648] HybridClock initialized: now 1790313190378158 us; error 38 us; skew 500 ppm
Sep 25 05:13:10 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313190, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:10.381402 18784 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:13:10.381994 18784 webserver.cc:460] Webserver started at http://127.3.6.131:33251/ using document root <none> and password file <none>
I20260925 05:13:10.382278 18784 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:10.382355 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:10.382516 18784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:13:10.383679 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "feb4987c963843b7a807ed3274da0b65"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a7c1161ad1451a0f759235454e3101da"
server_key_iv: "f3903d3f86cb6d51da6e5552b3ec2a2f"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.384131 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "feb4987c963843b7a807ed3274da0b65"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a7c1161ad1451a0f759235454e3101da"
server_key_iv: "f3903d3f86cb6d51da6e5552b3ec2a2f"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.385579 18784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:10.386435 18799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.386608 18784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:10.386729 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "feb4987c963843b7a807ed3274da0b65"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a7c1161ad1451a0f759235454e3101da"
server_key_iv: "f3903d3f86cb6d51da6e5552b3ec2a2f"
server_key_version: "encryptionkey@0"
I20260925 05:13:10.386806 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:10.401918 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:10.404477 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:10.404675 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:10.404934 18784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:10.405267 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:13:10.405323 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.405368 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:13:10.405392 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:10.411414 18784 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:45153
I20260925 05:13:10.411496 18912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:45153 every 8 connection(s)
I20260925 05:13:10.412000 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:13:10.416832 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 18784
I20260925 05:13:10.416950 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:13:10.417156 3098 external_mini_cluster.cc:1428] Setting key 8deb3c30fb6f30255fb81f6f641b2bf0
Sep 25 05:13:10 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313190, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:13:10.422284 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.412874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:44193 (local address 127.3.6.190:32979)
0925 05:13:10.412963 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.412965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.413203 (+ 238us) server_negotiation.cc:407] Connection header received
0925 05:13:10.413725 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.413727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.413738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.413825 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:10.414826 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.415321 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.416437 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.416627 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.416685 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.419478 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:10.419480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:10.419480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:10.419487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:10.420928 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.421336 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.421337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.421337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.421396 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.421635 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.421636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.421636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.421753 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:10.421809 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.422092 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.422188 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:13:10.422526 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.412846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:44193)
0925 05:13:10.413027 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.413074 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.413597 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.413893 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.413896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.414519 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:10.414729 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.414735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.415440 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.415441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.416357 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.416359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.416719 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.416720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.416886 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.417585 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:10.417597 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:10.419387 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:10.421004 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:10.421010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:10.421012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:10.421257 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:10.421462 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:10.421463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:10.421463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:10.421571 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:10.421811 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:10.421815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:10.421945 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.422063 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.422238 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260925 05:13:10.423126 18913 heartbeater.cc:347] Connected to a master server at 127.3.6.190:32979
I20260925 05:13:10.423252 18913 heartbeater.cc:464] Registering TS with master...
I20260925 05:13:10.423473 18913 heartbeater.cc:511] Master 127.3.6.190:32979 requested a full tablet report, sending...
I20260925 05:13:10.423962 18454 ts_manager.cc:194] Registered new tserver with Master: feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
I20260925 05:13:10.424369 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:44193
I20260925 05:13:10.429131 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:13:10.436594 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.430588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.3.6.190:32979)
0925 05:13:10.430661 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.430663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.430769 (+ 106us) server_negotiation.cc:407] Connection header received
0925 05:13:10.430828 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.430829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.430839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.430894 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:10.431464 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.432017 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.432722 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.432908 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.432959 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.434135 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:10.434136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:10.434137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:10.434143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:10.435589 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.435958 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.435959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.435959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.436012 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:10.436160 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:10.436160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:10.436160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:10.436300 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:10.436375 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.436469 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.436530 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:13:10.438347 18454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:13:10.439953 18454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:13:10.447283 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.442712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.3.6.131:45153)
0925 05:13:10.442975 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.442980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.442992 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:13:10.443031 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.443033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.443054 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.443142 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:10.443342 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.444607 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.446244 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.446755 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.446880 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.446953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.447127 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.447180 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260925 05:13:10.447377 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.442548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46609 (local address 127.0.0.1:59476)
0925 05:13:10.442698 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.442742 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.442823 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.443638 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.443639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.443641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:10.443676 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.443677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.444678 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.444679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.446111 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.446114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.447027 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.447028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.447112 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.447115 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.447169 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.447238 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":104,"thread_start_us":49,"threads_started":1}
I20260925 05:13:10.447367 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.442662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.3.6.130:46609)
0925 05:13:10.443444 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.443448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.443456 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:13:10.443514 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.443516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.443531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.443756 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:10.443881 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.444561 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.446183 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.446911 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.447021 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.447074 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.447183 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.447274 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":94,"threads_started":1}
I20260925 05:13:10.447398 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.442747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:37449 (local address 127.0.0.1:47626)
0925 05:13:10.443250 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.443261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.443314 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.443510 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.443511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.443514 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:10.443571 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.443573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.444858 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.444859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.446052 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.446055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.447089 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.447090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.447158 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.447164 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.447216 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.447271 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":5,"thread_start_us":74,"threads_started":1}
I20260925 05:13:10.447655 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.442995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.3.6.129:37449)
0925 05:13:10.443156 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.443160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.443323 (+ 163us) server_negotiation.cc:407] Connection header received
0925 05:13:10.443388 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.443391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.443406 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.443497 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:10.443655 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.444763 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.446122 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.446909 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.446989 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.447052 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.447498 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.447545 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260925 05:13:10.447703 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.442549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:45153 (local address 127.0.0.1:36738)
0925 05:13:10.442892 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.442903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.442949 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.443157 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.443161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.443163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:10.443236 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.443238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.444733 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.444734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.445768 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.445771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.446883 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.446885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.446988 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.447005 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.447079 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.447649 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":289,"thread_start_us":84,"threads_started":1}
I20260925 05:13:10.448400 18713 tablet_service.cc:1467] Processing CreateTablet for tablet beaaf1c42eab4b2a98eb268a6d69b79b (DEFAULT_TABLE table=test-table [id=6ee3c67a11434fa08ce9fd9e3fd5a422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:13:10.448485 18579 tablet_service.cc:1467] Processing CreateTablet for tablet beaaf1c42eab4b2a98eb268a6d69b79b (DEFAULT_TABLE table=test-table [id=6ee3c67a11434fa08ce9fd9e3fd5a422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:13:10.448861 18713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaaf1c42eab4b2a98eb268a6d69b79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:13:10.448880 18579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaaf1c42eab4b2a98eb268a6d69b79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:13:10.451829 18930 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Bootstrap starting.
I20260925 05:13:10.451886 18931 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Bootstrap starting.
I20260925 05:13:10.452495 18931 tablet_bootstrap.cc:654] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Neither blocks nor log segments found. Creating new log.
I20260925 05:13:10.452538 18930 tablet_bootstrap.cc:654] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Neither blocks nor log segments found. Creating new log.
I20260925 05:13:10.452458 18847 tablet_service.cc:1467] Processing CreateTablet for tablet beaaf1c42eab4b2a98eb268a6d69b79b (DEFAULT_TABLE table=test-table [id=6ee3c67a11434fa08ce9fd9e3fd5a422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:13:10.452853 18847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaaf1c42eab4b2a98eb268a6d69b79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:13:10.452941 18931 log.cc:824] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:10.453107 18930 log.cc:824] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:10.453631 18931 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: No bootstrap required, opened a new log
I20260925 05:13:10.453679 18931 ts_tablet_manager.cc:1397] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:13:10.455480 18931 raft_consensus.cc:348] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.455616 18931 raft_consensus.cc:374] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:13:10.455638 18931 raft_consensus.cc:729] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a751c234ae46d2923344de5d229a5b, State: Initialized, Role: FOLLOWER
I20260925 05:13:10.455783 18934 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Bootstrap starting.
I20260925 05:13:10.455756 18931 consensus_queue.cc:260] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.456072 18931 ts_tablet_manager.cc:1428] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:13:10.456223 18779 heartbeater.cc:503] Master 127.3.6.190:32979 was elected leader, sending a full tablet report...
I20260925 05:13:10.456607 18934 tablet_bootstrap.cc:654] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Neither blocks nor log segments found. Creating new log.
I20260925 05:13:10.457226 18934 log.cc:824] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:10.457253 18930 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: No bootstrap required, opened a new log
I20260925 05:13:10.457324 18930 ts_tablet_manager.cc:1397] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 05:13:10.458031 18934 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: No bootstrap required, opened a new log
I20260925 05:13:10.458104 18934 ts_tablet_manager.cc:1397] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:13:10.459476 18930 raft_consensus.cc:348] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.459667 18930 raft_consensus.cc:374] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:13:10.459712 18930 raft_consensus.cc:729] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a054633e36c468d83130306818630a1, State: Initialized, Role: FOLLOWER
I20260925 05:13:10.459908 18930 consensus_queue.cc:260] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.460069 18934 raft_consensus.cc:348] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.460265 18645 heartbeater.cc:503] Master 127.3.6.190:32979 was elected leader, sending a full tablet report...
I20260925 05:13:10.460286 18934 raft_consensus.cc:374] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:13:10.460248 18930 ts_tablet_manager.cc:1428] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:13:10.460347 18934 raft_consensus.cc:729] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb4987c963843b7a807ed3274da0b65, State: Initialized, Role: FOLLOWER
I20260925 05:13:10.460487 18934 consensus_queue.cc:260] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.460803 18913 heartbeater.cc:503] Master 127.3.6.190:32979 was elected leader, sending a full tablet report...
I20260925 05:13:10.461073 18934 ts_tablet_manager.cc:1428] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260925 05:13:10.490682 18780 tablet.cc:2369] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:13:10.504738 18937 raft_consensus.cc:482] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:13:10.504916 18937 raft_consensus.cc:504] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.505336 18937 leader_election.cc:288] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
I20260925 05:13:10.508381 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.505523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46609 (local address 127.3.6.129:41315)
0925 05:13:10.505586 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.505598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.505640 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.505838 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.505840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.505842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:10.505921 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.505923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.506649 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.506651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.507496 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.507498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.508082 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.508083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.508177 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.508181 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.508235 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.508267 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":44}
I20260925 05:13:10.508431 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.505637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:49673 (local address 127.3.6.131:45153)
0925 05:13:10.505698 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.505700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.505837 (+ 137us) server_negotiation.cc:407] Connection header received
0925 05:13:10.505902 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.505904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.505918 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.505982 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:10.506150 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.506630 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.507607 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.508075 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.508149 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.508160 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.508315 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.508356 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:13:10.508409 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.505598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:41315 (local address 127.3.6.130:46609)
0925 05:13:10.505663 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.505665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.505694 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:13:10.505723 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.505724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.505738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.505799 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:10.506008 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.506556 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.507597 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.507998 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.508053 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.508062 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.508254 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.508330 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:13:10.508497 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.505535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:45153 (local address 127.3.6.129:49673)
0925 05:13:10.505746 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.505756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:10.505832 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:10.506008 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:10.506008 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:10.506010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:10.506077 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.506079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.506721 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.506722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.507502 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:10.507504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.508160 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.508161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:10.508232 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.508238 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:13:10.508298 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:13:10.508337 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":16,"thread_start_us":119,"threads_started":1}
I20260925 05:13:10.508744 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaaf1c42eab4b2a98eb268a6d69b79b" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a751c234ae46d2923344de5d229a5b" is_pre_election: true
I20260925 05:13:10.508836 18867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaaf1c42eab4b2a98eb268a6d69b79b" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb4987c963843b7a807ed3274da0b65" is_pre_election: true
I20260925 05:13:10.508919 18733 raft_consensus.cc:2454] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 0.
I20260925 05:13:10.508962 18867 raft_consensus.cc:2454] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 0.
I20260925 05:13:10.509193 18535 leader_election.cc:302] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a054633e36c468d83130306818630a1, feb4987c963843b7a807ed3274da0b65; no voters:
I20260925 05:13:10.509377 18937 raft_consensus.cc:2781] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:13:10.509454 18937 raft_consensus.cc:482] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:13:10.509472 18937 raft_consensus.cc:3037] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:13:10.510279 18937 raft_consensus.cc:504] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.510432 18937 leader_election.cc:288] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 election: Requested vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
I20260925 05:13:10.510596 18867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaaf1c42eab4b2a98eb268a6d69b79b" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb4987c963843b7a807ed3274da0b65"
I20260925 05:13:10.510639 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaaf1c42eab4b2a98eb268a6d69b79b" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a751c234ae46d2923344de5d229a5b"
I20260925 05:13:10.510715 18733 raft_consensus.cc:3037] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:13:10.510715 18867 raft_consensus.cc:3037] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:13:10.511690 18733 raft_consensus.cc:2454] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 1.
I20260925 05:13:10.511690 18867 raft_consensus.cc:2454] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 1.
I20260925 05:13:10.511919 18534 leader_election.cc:302] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a751c234ae46d2923344de5d229a5b, 9a054633e36c468d83130306818630a1; no voters:
I20260925 05:13:10.512048 18937 raft_consensus.cc:2781] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:13:10.512212 18937 raft_consensus.cc:686] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 1 LEADER]: Becoming Leader. State: Replica: 9a054633e36c468d83130306818630a1, State: Running, Role: LEADER
I20260925 05:13:10.512320 18937 consensus_queue.cc:237] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:10.513319 18453 catalog_manager.cc:5495] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a054633e36c468d83130306818630a1 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "9a054633e36c468d83130306818630a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } health_report { overall_health: UNKNOWN } } }
I20260925 05:13:10.521579 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.518909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.3.6.129:37449)
0925 05:13:10.518962 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:10.518964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:10.518974 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:13:10.519013 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:10.519014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:10.519043 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:10.519122 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:13:10.519278 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.519967 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.520761 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:10.520982 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:10.521061 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:10.521233 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:13:10.521350 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:13:10.521449 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:10.521469 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 05:13:10.521490 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:13:10.523324 18867 raft_consensus.cc:1261] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Refusing update from remote peer 9a054633e36c468d83130306818630a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:13:10.523383 18733 raft_consensus.cc:1261] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Refusing update from remote peer 9a054633e36c468d83130306818630a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:13:10.523695 18940 consensus_queue.cc:1035] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:13:10.523932 18940 consensus_queue.cc:1035] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:13:10.526578 18945 mvcc.cc:205] Tried to move back new op lower bound from 7333122828380991488 to 7333122828339191808. Current Snapshot: MvccSnapshot[applied={T|T < 7333122828380991488}]
I20260925 05:13:10.527061 18944 mvcc.cc:205] Tried to move back new op lower bound from 7333122828380991488 to 7333122828339191808. Current Snapshot: MvccSnapshot[applied={T|T < 7333122828380991488 or (T in {7333122828380991488})}]
I20260925 05:13:10.529301 18453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:13:10.529429 18453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:13:10.530264 18453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Sending DeleteTablet for 3 replicas of tablet beaaf1c42eab4b2a98eb268a6d69b79b
I20260925 05:13:10.530733 18579 tablet_service.cc:1514] Processing DeleteTablet for tablet beaaf1c42eab4b2a98eb268a6d69b79b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:13:10 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:47626
I20260925 05:13:10.530740 18847 tablet_service.cc:1514] Processing DeleteTablet for tablet beaaf1c42eab4b2a98eb268a6d69b79b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:13:10 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:36738
I20260925 05:13:10.531013 18713 tablet_service.cc:1514] Processing DeleteTablet for tablet beaaf1c42eab4b2a98eb268a6d69b79b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:13:10 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:59476
I20260925 05:13:10.531046 18954 tablet_replica.cc:330] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: stopping tablet replica
I20260925 05:13:10.531174 18954 raft_consensus.cc:2229] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:13:10.531279 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 18516
I20260925 05:13:10.531311 18954 raft_consensus.cc:2258] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:13:10.531315 18955 tablet_replica.cc:330] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: stopping tablet replica
I20260925 05:13:10.531430 18955 raft_consensus.cc:2229] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:13:10.531535 18955 raft_consensus.cc:2258] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:13:10.532303 18954 ts_tablet_manager.cc:1905] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:13:10.532383 18955 ts_tablet_manager.cc:1905] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:13:10.533864 18954 ts_tablet_manager.cc:1918] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:13:10.533915 18954 log.cc:1196] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/beaaf1c42eab4b2a98eb268a6d69b79b
I20260925 05:13:10.534149 18954 ts_tablet_manager.cc:1939] T beaaf1c42eab4b2a98eb268a6d69b79b P feb4987c963843b7a807ed3274da0b65: Deleting consensus metadata
I20260925 05:13:10.534440 18955 ts_tablet_manager.cc:1918] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:13:10.534509 18955 log.cc:1196] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/beaaf1c42eab4b2a98eb268a6d69b79b
I20260925 05:13:10.534535 18441 catalog_manager.cc:4841] TS feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): tablet beaaf1c42eab4b2a98eb268a6d69b79b (table test-table [id=6ee3c67a11434fa08ce9fd9e3fd5a422]) successfully deleted
I20260925 05:13:10.534780 18955 ts_tablet_manager.cc:1939] T beaaf1c42eab4b2a98eb268a6d69b79b P 10a751c234ae46d2923344de5d229a5b: Deleting consensus metadata
I20260925 05:13:10.535260 18440 catalog_manager.cc:4841] TS 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): tablet beaaf1c42eab4b2a98eb268a6d69b79b (table test-table [id=6ee3c67a11434fa08ce9fd9e3fd5a422]) successfully deleted
W20260925 05:13:10.538587 18440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:37449 (error 108)
I20260925 05:13:10.538903 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 18650
I20260925 05:13:10.539552 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:10.539068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:37449 (local address 127.0.0.1:47638)
0925 05:13:10.539463 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:10.539507 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:37449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
W20260925 05:13:10.539703 18440 catalog_manager.cc:4584] TS 9a054633e36c468d83130306818630a1 (127.3.6.129:37449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet beaaf1c42eab4b2a98eb268a6d69b79b: Network error: Client connection negotiation failed: client connection to 127.3.6.129:37449: connect: Connection refused (error 111)
I20260925 05:13:10.544065 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 18784
I20260925 05:13:10.549532 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 18422
I20260925 05:13:30.555497 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32979
--webserver_interface=127.3.6.190
--webserver_port=37407
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:32979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:13:30.639878 18957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:30.640111 18957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:30.640136 18957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:30.641273 18957 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:13:30.641305 18957 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:13:30.641319 18957 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:13:30.641330 18957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:13:30.641343 18957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:30.641356 18957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:30.641368 18957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:13:30.641386 18957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:13:30.642820 18957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:32979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:30.643029 18957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:30.643224 18957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:30.652669 18963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:30.652825 18957 server_base.cc:1034] running on GCE node
W20260925 05:13:30.652925 18964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:30.653071 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:30.653424 18957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:30.653810 18957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:30.655066 18957 hybrid_clock.cc:648] HybridClock initialized: now 1790313210655047 us; error 46 us; skew 500 ppm
Sep 25 05:13:30 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313210, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:30.664654 18957 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:13:30.665270 18957 webserver.cc:460] Webserver started at http://127.3.6.190:37407/ using document root <none> and password file <none>
I20260925 05:13:30.665539 18957 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:30.665634 18957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:30.667157 18957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:13:30.667936 18973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:30.668116 18957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:13:30.668226 18957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "9a8b78fa59d14e4586b2f6597e793c08"
format_stamp: "Formatted at 2026-09-25 05:13:09 on dist-test-slave-1lvv"
server_key: "6110eaec16ba8fcc59770bcfedc4860d"
server_key_iv: "a64c6891d2d5f82edc8161b8c6dad477"
server_key_version: "encryptionkey@0"
I20260925 05:13:30.668664 18957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:30.675832 18957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:30.678292 18957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:30.678467 18957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:30.682858 18957 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:32979
I20260925 05:13:30.682893 19025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:32979 every 8 connection(s)
I20260925 05:13:30.683306 18957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:13:30.684880 19026 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:13:30.685575 19026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Bootstrap starting.
I20260925 05:13:30.691785 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 18957
I20260925 05:13:30.692142 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:37449
--local_ip_for_outbound_sockets=127.3.6.129
--tserver_master_addrs=127.3.6.190:32979
--webserver_port=44953
--webserver_interface=127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:13:30.695019 19026 log.cc:824] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:30.697397 19026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:13:30.697742 19026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Bootstrap complete.
I20260925 05:13:30.700284 19026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:30.700531 19026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8b78fa59d14e4586b2f6597e793c08, State: Initialized, Role: FOLLOWER
I20260925 05:13:30.700707 19026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:30.700786 19026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:13:30.700840 19026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:13:30.700893 19026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:13:30.702045 19026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:30.702139 19026 leader_election.cc:302] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a8b78fa59d14e4586b2f6597e793c08; no voters:
I20260925 05:13:30.702368 19026 leader_election.cc:288] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:13:30.702414 19030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:13:30.702605 19030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 LEADER]: Becoming Leader. State: Replica: 9a8b78fa59d14e4586b2f6597e793c08, State: Running, Role: LEADER
I20260925 05:13:30.702674 19026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:13:30.702804 19030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:30.703199 19030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8b78fa59d14e4586b2f6597e793c08. Latest consensus state: current_term: 2 leader_uuid: "9a8b78fa59d14e4586b2f6597e793c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } } }
I20260925 05:13:30.703204 19032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a8b78fa59d14e4586b2f6597e793c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } } }
I20260925 05:13:30.703429 19032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: This master's current role is: LEADER
I20260925 05:13:30.703807 19030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: This master's current role is: LEADER
I20260925 05:13:30.703977 19039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:13:30.704999 19039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:13:30.705112 19039 catalog_manager.cc:1259] Loaded cluster ID: 5236ea3a1e794dcd94d5d229895ad202
I20260925 05:13:30.705147 19039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:13:30.706338 19039 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:13:30.706528 19039 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Loaded TSK: 0
I20260925 05:13:30.724566 19039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Generated new TSK 1
I20260925 05:13:30.724694 19039 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:13:30.779443 19028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:30.779716 19028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:30.779764 19028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:30.781030 19028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:30.781104 19028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:30.781148 19028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:13:30.782691 19028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:37449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=44953
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:30.782967 19028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:30.783222 19028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:30.788481 19053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:30.788489 19056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:30.788690 19028 server_base.cc:1034] running on GCE node
W20260925 05:13:30.788470 19054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:30.789019 19028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:30.789335 19028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:30.790539 19028 hybrid_clock.cc:648] HybridClock initialized: now 1790313210790530 us; error 37 us; skew 500 ppm
Sep 25 05:13:30 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313210, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:30.793797 19028 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:13:30.794284 19028 webserver.cc:460] Webserver started at http://127.3.6.129:44953/ using document root <none> and password file <none>
I20260925 05:13:30.794550 19028 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:30.794656 19028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:30.796229 19028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:13:30.797076 19063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:30.797313 19028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:13:30.797429 19028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "9a054633e36c468d83130306818630a1"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "693a43691d90e6209a6a8f3537167505"
server_key_iv: "8ab3ca97c88c662db90b08f0ba87c88f"
server_key_version: "encryptionkey@0"
I20260925 05:13:30.797828 19028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:30.806588 19028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:30.809170 19028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:30.809341 19028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:30.809607 19028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:30.810092 19070 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:13:30.812283 19028 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:13:30.812368 19028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 05:13:30.812423 19028 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:13:30.813089 19028 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:13:30.813149 19028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 05:13:30.813221 19070 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Bootstrap starting.
I20260925 05:13:30.820811 19028 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:37449
I20260925 05:13:30.820852 19177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:37449 every 8 connection(s)
I20260925 05:13:30.821264 19028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:13:30.824445 19070 log.cc:824] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:30.827564 19070 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:13:30.828011 19070 tablet_bootstrap.cc:492] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Bootstrap complete.
I20260925 05:13:30.828881 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 19028
I20260925 05:13:30.829237 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:46609
--local_ip_for_outbound_sockets=127.3.6.130
--tserver_master_addrs=127.3.6.190:32979
--webserver_port=45387
--webserver_interface=127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:13:30.829344 19070 ts_tablet_manager.cc:1397] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
Sep 25 05:13:30 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313210, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:13:30.832664 19070 raft_consensus.cc:348] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:30.833448 19070 raft_consensus.cc:729] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a054633e36c468d83130306818630a1, State: Initialized, Role: FOLLOWER
I20260925 05:13:30.833701 19070 consensus_queue.cc:260] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } }
I20260925 05:13:30.834177 19070 ts_tablet_manager.cc:1428] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:13:30.836632 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:30.822051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:43563)
0925 05:13:30.823952 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:30.823992 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:30.824475 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:30.825502 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:30.825507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:30.825824 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:30.826014 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:30.826020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.827228 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.827229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:30.828240 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:30.828243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.828987 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.828988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:30.829182 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:30.829860 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:30.829870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:30.831865 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:30.834444 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:30.834451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:30.834453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:30.834815 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:30.835151 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:30.835152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:30.835153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:30.835308 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:30.835918 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:30.835926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:30.836100 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 05:13:30.836222 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 05:13:30.836409 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1855,"thread_start_us":92,"threads_started":1}
I20260925 05:13:30.836808 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:30.822138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:43563 (local address 127.3.6.190:32979)
0925 05:13:30.825120 (+ 2982us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:30.825128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:30.825160 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:13:30.825274 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:30.825279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:30.825321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:30.825468 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:30.826141 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.827097 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.828334 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.828856 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.829501 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:30.832007 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:30.832013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:30.832013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:30.832027 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:30.834338 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:30.834926 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:30.834929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:30.834930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:30.835060 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:30.835417 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:30.835418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:30.835419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:30.835795 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:30.835920 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:30.836330 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 05:13:30.836536 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2931,"thread_start_us":73,"threads_started":1}
I20260925 05:13:30.837862 19178 heartbeater.cc:347] Connected to a master server at 127.3.6.190:32979
I20260925 05:13:30.837975 19178 heartbeater.cc:464] Registering TS with master...
I20260925 05:13:30.838184 19178 heartbeater.cc:511] Master 127.3.6.190:32979 requested a full tablet report, sending...
I20260925 05:13:30.839061 18990 ts_manager.cc:194] Registered new tserver with Master: 9a054633e36c468d83130306818630a1 (127.3.6.129:37449)
I20260925 05:13:30.840307 18990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:43563
I20260925 05:13:30.841953 19178 heartbeater.cc:503] Master 127.3.6.190:32979 was elected leader, sending a full tablet report...
Sep 25 05:13:30 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313210, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.129@KRBTEST.COM
I20260925 05:13:30.852340 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:30.839734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:37449 (local address 127.0.0.1:45228)
0925 05:13:30.840339 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:30.840380 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:30.840459 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:30.840937 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:30.840943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:30.841336 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:30.841435 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:30.841437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.843155 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.843157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:30.844207 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:30.844211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.844909 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.844911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:30.845048 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:30.846248 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:30.846253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:30.848461 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:30.850856 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:30.850863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:30.850866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:30.851174 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:30.851444 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:30.851445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:30.851445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:30.851529 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:30.851913 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:30.851919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:30.851981 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 05:13:30.852078 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:13:30.852218 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":377,"threads_started":1}
I20260925 05:13:30.852353 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:30.839892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.3.6.129:37449)
0925 05:13:30.840264 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:30.840268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:30.840762 (+ 494us) server_negotiation.cc:407] Connection header received
0925 05:13:30.840812 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:30.840814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:30.840830 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:30.840900 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:30.841866 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.843042 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.844302 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.844812 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.844889 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:30.848579 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:30.848590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:30.848591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:30.848600 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:30.850752 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:30.851290 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:30.851294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:30.851294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:30.851363 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:30.851616 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:30.851617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:30.851617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:30.851851 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:30.851935 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:30.852143 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 05:13:30.852235 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":91,"threads_started":1}
I20260925 05:13:30.853194 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet beaaf1c42eab4b2a98eb268a6d69b79b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:13:10 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45228
I20260925 05:13:30.853996 19187 tablet_replica.cc:330] stopping tablet replica
I20260925 05:13:30.854126 19187 raft_consensus.cc:2229] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:13:30.854208 19187 raft_consensus.cc:2258] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:13:30.854598 19187 ts_tablet_manager.cc:1905] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:13:30.857496 19187 ts_tablet_manager.cc:1918] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:13:30.857595 19187 log.cc:1196] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal/wals/beaaf1c42eab4b2a98eb268a6d69b79b
I20260925 05:13:30.857929 19187 ts_tablet_manager.cc:1939] T beaaf1c42eab4b2a98eb268a6d69b79b P 9a054633e36c468d83130306818630a1: Deleting consensus metadata
I20260925 05:13:30.858587 18976 catalog_manager.cc:4841] TS 9a054633e36c468d83130306818630a1 (127.3.6.129:37449): tablet beaaf1c42eab4b2a98eb268a6d69b79b (table test-table [id=6ee3c67a11434fa08ce9fd9e3fd5a422]) successfully deleted
W20260925 05:13:30.920903 19183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:30.921144 19183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:30.921168 19183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:30.922371 19183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:30.922418 19183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:30.922447 19183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:13:30.923985 19183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:46609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=45387
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:30.924248 19183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:30.924492 19183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:30.929893 19192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:30.929884 19193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:30.929886 19195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:30.930193 19183 server_base.cc:1034] running on GCE node
I20260925 05:13:30.930486 19183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:30.930785 19183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:30.931931 19183 hybrid_clock.cc:648] HybridClock initialized: now 1790313210931919 us; error 31 us; skew 500 ppm
Sep 25 05:13:30 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313210, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:30.934499 19183 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:13:30.934919 19183 webserver.cc:460] Webserver started at http://127.3.6.130:45387/ using document root <none> and password file <none>
I20260925 05:13:30.935107 19183 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:30.935163 19183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:30.936595 19183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:30.937178 19202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:30.937373 19183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:30.937460 19183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "10a751c234ae46d2923344de5d229a5b"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a0b7c7739a20e135f487ba5e7738db68"
server_key_iv: "d3674918fedab1ac625d764391fa8c3e"
server_key_version: "encryptionkey@0"
I20260925 05:13:30.937810 19183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:30.944804 19183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:30.947154 19183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:30.947288 19183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:30.947503 19183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:30.947929 19183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:13:30.947973 19183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:30.947995 19183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:13:30.948040 19183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:30.953910 19183 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:46609
I20260925 05:13:30.954021 19315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:46609 every 8 connection(s)
I20260925 05:13:30.954345 19183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:13:30.956908 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 19183
I20260925 05:13:30.957213 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:45153
--local_ip_for_outbound_sockets=127.3.6.131
--tserver_master_addrs=127.3.6.190:32979
--webserver_port=33251
--webserver_interface=127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:13:30 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313210, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:13:30.965216 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:30.955177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:48739 (local address 127.3.6.190:32979)
0925 05:13:30.955226 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:30.955228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:30.955483 (+ 255us) server_negotiation.cc:407] Connection header received
0925 05:13:30.956096 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:30.956098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:30.956115 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:30.956192 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:30.957219 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.957801 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.959056 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.959298 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.959371 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:30.962038 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:30.962040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:30.962041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:30.962057 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:30.963600 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:30.964123 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:30.964125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:30.964125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:30.964191 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:30.964480 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:30.964481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:30.964481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:30.964611 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:30.964694 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:30.965006 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 05:13:30.965110 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:13:30.965417 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:30.955161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:48739)
0925 05:13:30.955345 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:30.955384 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:30.955971 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:30.956275 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:30.956278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:30.956863 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:30.957120 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:30.957126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.957902 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.957903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:30.958943 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:30.958946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:30.959396 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:30.959397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:30.959548 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:30.960171 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:30.960182 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:30.961909 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:30.963723 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:30.963733 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:30.963735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:30.964028 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:30.964275 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:30.964276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:30.964276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:30.964404 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:30.964684 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:30.964692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:30.964859 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 05:13:30.964972 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:13:30.965156 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260925 05:13:30.965950 19316 heartbeater.cc:347] Connected to a master server at 127.3.6.190:32979
I20260925 05:13:30.966094 19316 heartbeater.cc:464] Registering TS with master...
I20260925 05:13:30.966327 19316 heartbeater.cc:511] Master 127.3.6.190:32979 requested a full tablet report, sending...
I20260925 05:13:30.966816 18990 ts_manager.cc:194] Registered new tserver with Master: 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609)
I20260925 05:13:30.967242 18990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:48739
W20260925 05:13:31.044795 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:31.045095 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:31.045142 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:31.046386 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:31.046470 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:31.046519 19319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:13:31.048236 19319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:45153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=33251
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:31.048553 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:31.048808 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:31.053952 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:31.053947 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:31.054174 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:31.054145 19319 server_base.cc:1034] running on GCE node
I20260925 05:13:31.054517 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:31.054771 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:31.055918 19319 hybrid_clock.cc:648] HybridClock initialized: now 1790313211055902 us; error 39 us; skew 500 ppm
Sep 25 05:13:31 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313211, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:31.062450 19319 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:13:31.062995 19319 webserver.cc:460] Webserver started at http://127.3.6.131:33251/ using document root <none> and password file <none>
I20260925 05:13:31.063275 19319 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:31.063337 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:31.066277 19319 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:13:31.067009 19334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:31.067188 19319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:31.067276 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "feb4987c963843b7a807ed3274da0b65"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a7c1161ad1451a0f759235454e3101da"
server_key_iv: "f3903d3f86cb6d51da6e5552b3ec2a2f"
server_key_version: "encryptionkey@0"
I20260925 05:13:31.067618 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:31.078562 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:31.081033 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:31.081166 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:31.081367 19319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:31.081705 19319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:13:31.081743 19319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:31.081790 19319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:13:31.081817 19319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:31.087604 19319 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:45153
I20260925 05:13:31.087694 19447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:45153 every 8 connection(s)
I20260925 05:13:31.088083 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:13:31.093771 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 19319
Sep 25 05:13:31 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313211, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:13:31.098003 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.088880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:39055 (local address 127.3.6.190:32979)
0925 05:13:31.088956 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.088958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.089171 (+ 213us) server_negotiation.cc:407] Connection header received
0925 05:13:31.089671 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.089672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.089683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.089740 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:31.090690 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.091180 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.092242 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.092428 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.092484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.094783 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:31.094784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:31.094785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:31.094793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:31.096480 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:31.096901 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:31.096902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:31.096903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:31.096970 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:31.097265 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:31.097266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:31.097267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:31.097390 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:31.097476 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.097784 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.097884 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:13:31.098191 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.088868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:39055)
0925 05:13:31.089030 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:31.089067 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:31.089580 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:31.089808 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:31.089811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:31.090403 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:31.090603 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.090607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.091260 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.091261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.092167 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.092170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.092498 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.092499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.092597 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.093073 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:31.093081 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:31.094712 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:31.096567 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:31.096575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:31.096577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:31.096829 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:31.097046 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:31.097047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:31.097047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:31.097170 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:31.097466 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:31.097473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:31.097626 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 05:13:31.097751 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 05:13:31.097943 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260925 05:13:31.098869 19448 heartbeater.cc:347] Connected to a master server at 127.3.6.190:32979
I20260925 05:13:31.098974 19448 heartbeater.cc:464] Registering TS with master...
I20260925 05:13:31.099180 19448 heartbeater.cc:511] Master 127.3.6.190:32979 requested a full tablet report, sending...
I20260925 05:13:31.099627 18990 ts_manager.cc:194] Registered new tserver with Master: feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
I20260925 05:13:31.100232 18990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:39055
I20260925 05:13:31.102994 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.094230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.3.6.190:32979)
0925 05:13:31.094367 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.094369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.094538 (+ 169us) server_negotiation.cc:407] Connection header received
0925 05:13:31.094613 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.094614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.094627 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.094680 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:31.095482 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.096149 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.097183 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.097398 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.097472 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.099436 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:31.099438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:31.099438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:31.099446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:31.101435 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:31.101902 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:31.101903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:31.101904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:31.101973 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:31.102227 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:31.102229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:31.102229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:31.102359 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:31.102551 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.102765 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.102856 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260925 05:13:31.103796 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:13:31.106436 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.104148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.3.6.190:32979)
0925 05:13:31.104202 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.104204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.104415 (+ 211us) server_negotiation.cc:407] Connection header received
0925 05:13:31.104443 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.104444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.104470 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.104520 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:13:31.104644 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.105081 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.105802 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.105985 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.106035 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.106202 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:13:31.106256 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 05:13:31.106394 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60190: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 05:13:31.106510 19451 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60190: token expired
W20260925 05:13:31.106895 18919 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.6.190:32979)
I20260925 05:13:31.114424 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.107238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.3.6.190:32979)
0925 05:13:31.107297 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.107299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.107306 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:13:31.107326 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.107327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.107335 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.107380 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:31.107893 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.108350 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.109165 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.109372 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.109447 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.111170 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:31.111172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:31.111172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:31.111180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:31.112991 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:31.113557 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:31.113558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:31.113558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:31.113626 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:31.113852 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:31.113853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:31.113854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:31.113986 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:31.114105 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.114276 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.114356 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:13:31.116039 18990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:13:31.117587 18990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:13:31.119942 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 875b5d0df5c44321b9d5bf81d706fae7 (DEFAULT_TABLE table=test-table [id=b559ac7a33aa4fbab7a423b02e3d8caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:13:31.120189 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875b5d0df5c44321b9d5bf81d706fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:13:31.121898 19070 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Bootstrap starting.
I20260925 05:13:31.122354 19070 tablet_bootstrap.cc:654] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Neither blocks nor log segments found. Creating new log.
I20260925 05:13:31.123363 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.119480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46609 (local address 127.0.0.1:60244)
0925 05:13:31.119612 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:31.119623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:31.119662 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:31.120020 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:31.120021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:31.120023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:31.120080 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.120082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.121306 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.121308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.122289 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.122293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.123082 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.123083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.123198 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.123226 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 05:13:31.123263 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 05:13:31.123310 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 05:13:31.123394 19070 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: No bootstrap required, opened a new log
I20260925 05:13:31.123442 19070 ts_tablet_manager.cc:1397] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:13:31.123594 19070 raft_consensus.cc:348] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.123672 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.119479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:45153 (local address 127.0.0.1:52904)
0925 05:13:31.119670 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:31.119679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:31.119721 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:31.120233 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:31.120233 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:31.120236 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:31.120270 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.120271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.121362 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.121363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.122426 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.122429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.123421 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.123422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.123490 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.123495 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:13:31.123559 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:13:31.123595 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":22,"thread_start_us":109,"threads_started":1}
I20260925 05:13:31.123664 19070 raft_consensus.cc:374] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:13:31.123713 19070 raft_consensus.cc:729] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a054633e36c468d83130306818630a1, State: Initialized, Role: FOLLOWER
I20260925 05:13:31.123705 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.119799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.3.6.131:45153)
0925 05:13:31.119987 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.119992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.120003 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:13:31.120117 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.120120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.120138 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.120391 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:31.120519 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.121288 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.122529 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.123328 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.123409 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.123478 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.123575 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.123621 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260925 05:13:31.123791 19070 consensus_queue.cc:260] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.123900 19070 ts_tablet_manager.cc:1428] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:31.123900 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.119604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.3.6.130:46609)
0925 05:13:31.119799 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.119804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.119816 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:13:31.119860 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.119862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.119879 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.120005 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:31.120152 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.121206 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.122386 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.122951 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.123556 (+ 605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.123630 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.123657 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.123705 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260925 05:13:31.124691 19382 tablet_service.cc:1467] Processing CreateTablet for tablet 875b5d0df5c44321b9d5bf81d706fae7 (DEFAULT_TABLE table=test-table [id=b559ac7a33aa4fbab7a423b02e3d8caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:13:31.124691 19250 tablet_service.cc:1467] Processing CreateTablet for tablet 875b5d0df5c44321b9d5bf81d706fae7 (DEFAULT_TABLE table=test-table [id=b559ac7a33aa4fbab7a423b02e3d8caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:13:31.125133 19382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875b5d0df5c44321b9d5bf81d706fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:13:31.125133 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875b5d0df5c44321b9d5bf81d706fae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:13:31.129213 19458 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Bootstrap starting.
I20260925 05:13:31.129963 19458 tablet_bootstrap.cc:654] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Neither blocks nor log segments found. Creating new log.
I20260925 05:13:31.130513 19458 log.cc:824] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:31.130766 19459 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Bootstrap starting.
I20260925 05:13:31.131335 19459 tablet_bootstrap.cc:654] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Neither blocks nor log segments found. Creating new log.
I20260925 05:13:31.131477 19458 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: No bootstrap required, opened a new log
I20260925 05:13:31.131574 19458 ts_tablet_manager.cc:1397] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:13:31.132009 19459 log.cc:824] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:31.132781 19459 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: No bootstrap required, opened a new log
I20260925 05:13:31.132853 19459 ts_tablet_manager.cc:1397] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 05:13:31.133539 19458 raft_consensus.cc:348] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.133713 19458 raft_consensus.cc:374] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:13:31.133757 19458 raft_consensus.cc:729] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb4987c963843b7a807ed3274da0b65, State: Initialized, Role: FOLLOWER
I20260925 05:13:31.133893 19458 consensus_queue.cc:260] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.134207 19458 ts_tablet_manager.cc:1428] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:13:31.134207 19448 heartbeater.cc:503] Master 127.3.6.190:32979 was elected leader, sending a full tablet report...
I20260925 05:13:31.134358 19459 raft_consensus.cc:348] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.134488 19459 raft_consensus.cc:374] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:13:31.134549 19459 raft_consensus.cc:729] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a751c234ae46d2923344de5d229a5b, State: Initialized, Role: FOLLOWER
I20260925 05:13:31.134673 19459 consensus_queue.cc:260] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.134984 19459 ts_tablet_manager.cc:1428] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:13:31.135005 19316 heartbeater.cc:503] Master 127.3.6.190:32979 was elected leader, sending a full tablet report...
I20260925 05:13:31.190227 19184 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:13:31.190428 19184 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.190922 19184 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
I20260925 05:13:31.194634 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.191165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:45153 (local address 127.3.6.129:50191)
0925 05:13:31.191230 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:31.191242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:31.191301 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:31.191509 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:31.191510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:31.191512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:31.191589 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.191594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.192622 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.192623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.193649 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.193652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.194287 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.194288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.194369 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.194421 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 05:13:31.194479 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:13:31.194526 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:13:31.195046 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.191226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:50191 (local address 127.3.6.131:45153)
0925 05:13:31.191293 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.191295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.191311 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:13:31.191418 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.191419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.191433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.191504 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:31.191692 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.192515 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.193775 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.194190 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.194288 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.194299 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.194930 (+ 631us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.194979 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:13:31.195410 19402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875b5d0df5c44321b9d5bf81d706fae7" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb4987c963843b7a807ed3274da0b65" is_pre_election: true
I20260925 05:13:31.195605 19402 raft_consensus.cc:2454] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 0.
I20260925 05:13:31.195673 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.191489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:58871 (local address 127.3.6.130:46609)
0925 05:13:31.191554 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.191556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.191565 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:13:31.191593 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.191607 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.191619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.191677 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:13:31.191966 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.192729 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.194397 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.195043 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.195113 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.195126 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.195556 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.195605 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:13:31.195688 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.191219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46609 (local address 127.3.6.129:58871)
0925 05:13:31.191423 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:31.191434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:31.191482 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:31.191734 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:31.191735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:31.191738 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:13:31.191870 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.191873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.192822 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.192823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.194220 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:31.194224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.195382 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.195383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:31.195457 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.195461 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:13:31.195510 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:13:31.195579 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":125,"threads_started":1}
I20260925 05:13:31.195917 19067 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a054633e36c468d83130306818630a1, feb4987c963843b7a807ed3274da0b65; no voters:
I20260925 05:13:31.196085 19184 raft_consensus.cc:2781] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:13:31.196151 19184 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:13:31.196178 19184 raft_consensus.cc:3037] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:13:31.196161 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875b5d0df5c44321b9d5bf81d706fae7" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a751c234ae46d2923344de5d229a5b" is_pre_election: true
I20260925 05:13:31.196316 19270 raft_consensus.cc:2454] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 0.
I20260925 05:13:31.196974 19184 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.197139 19184 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 election: Requested vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
I20260925 05:13:31.197352 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875b5d0df5c44321b9d5bf81d706fae7" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a751c234ae46d2923344de5d229a5b"
I20260925 05:13:31.197357 19402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875b5d0df5c44321b9d5bf81d706fae7" candidate_uuid: "9a054633e36c468d83130306818630a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb4987c963843b7a807ed3274da0b65"
I20260925 05:13:31.197434 19402 raft_consensus.cc:3037] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:13:31.197434 19270 raft_consensus.cc:3037] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:13:31.198361 19270 raft_consensus.cc:2454] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 1.
I20260925 05:13:31.198423 19402 raft_consensus.cc:2454] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a054633e36c468d83130306818630a1 in term 1.
I20260925 05:13:31.198639 19066 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a751c234ae46d2923344de5d229a5b, 9a054633e36c468d83130306818630a1; no voters:
I20260925 05:13:31.198771 19184 raft_consensus.cc:2781] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:13:31.198994 19184 raft_consensus.cc:686] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 LEADER]: Becoming Leader. State: Replica: 9a054633e36c468d83130306818630a1, State: Running, Role: LEADER
I20260925 05:13:31.199141 19184 consensus_queue.cc:237] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:31.200052 18989 catalog_manager.cc:5495] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a054633e36c468d83130306818630a1 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "9a054633e36c468d83130306818630a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } health_report { overall_health: HEALTHY } } }
W20260925 05:13:31.204941 19317 tablet.cc:2369] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:13:31.218488 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.216160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.3.6.129:37449)
0925 05:13:31.216211 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:31.216213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:31.216222 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:13:31.216285 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:31.216286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:31.216308 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:31.216367 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:13:31.216473 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.216965 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.217713 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:31.217918 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:31.217970 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:31.218144 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:13:31.218251 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:13:31.218307 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:31.218349 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 05:13:31.218369 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:13:31.220228 19402 raft_consensus.cc:1261] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Refusing update from remote peer 9a054633e36c468d83130306818630a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:13:31.220248 19270 raft_consensus.cc:1261] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Refusing update from remote peer 9a054633e36c468d83130306818630a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:13:31.220629 19465 consensus_queue.cc:1035] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:13:31.220786 19184 consensus_queue.cc:1035] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:13:31.223134 19467 mvcc.cc:205] Tried to move back new op lower bound from 7333122913155104768 to 7333122913072455680. Current Snapshot: MvccSnapshot[applied={T|T < 7333122913155104768}]
I20260925 05:13:31.224189 19468 mvcc.cc:205] Tried to move back new op lower bound from 7333122913155104768 to 7333122913072455680. Current Snapshot: MvccSnapshot[applied={T|T < 7333122913155104768}]
I20260925 05:13:31.225185 19469 mvcc.cc:205] Tried to move back new op lower bound from 7333122913155104768 to 7333122913072455680. Current Snapshot: MvccSnapshot[applied={T|T < 7333122913155104768 or (T in {7333122913155104768})}]
I20260925 05:13:31.225342 18989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:13:31.225446 18989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:13:31.226306 18989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Sending DeleteTablet for 3 replicas of tablet 875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:13:31.226666 19382 tablet_service.cc:1514] Processing DeleteTablet for tablet 875b5d0df5c44321b9d5bf81d706fae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:13:31 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:52904
I20260925 05:13:31.226706 19250 tablet_service.cc:1514] Processing DeleteTablet for tablet 875b5d0df5c44321b9d5bf81d706fae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:13:31 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:60244
I20260925 05:13:31.226910 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 875b5d0df5c44321b9d5bf81d706fae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:13:31 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45228
I20260925 05:13:31.227012 19187 tablet_replica.cc:330] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: stopping tablet replica
I20260925 05:13:31.227068 19187 raft_consensus.cc:2229] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:13:31.227068 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 19028
I20260925 05:13:31.227164 19187 raft_consensus.cc:2258] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:13:31.228348 19477 tablet_replica.cc:330] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: stopping tablet replica
I20260925 05:13:31.228487 19477 raft_consensus.cc:2229] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:13:31.228612 19477 raft_consensus.cc:2258] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:13:31.228808 19478 tablet_replica.cc:330] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: stopping tablet replica
I20260925 05:13:31.228955 19478 raft_consensus.cc:2229] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:13:31.229086 19478 raft_consensus.cc:2258] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:13:31.229532 19478 ts_tablet_manager.cc:1905] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:13:31.229568 19477 ts_tablet_manager.cc:1905] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:13:31.231628 19478 ts_tablet_manager.cc:1918] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:13:31.231707 19478 log.cc:1196] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:13:31.232002 19478 ts_tablet_manager.cc:1939] T 875b5d0df5c44321b9d5bf81d706fae7 P feb4987c963843b7a807ed3274da0b65: Deleting consensus metadata
I20260925 05:13:31.232155 19477 ts_tablet_manager.cc:1918] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:13:31.232301 19477 log.cc:1196] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:13:31.232437 18977 catalog_manager.cc:4841] TS feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): tablet 875b5d0df5c44321b9d5bf81d706fae7 (table test-table [id=b559ac7a33aa4fbab7a423b02e3d8caf]) successfully deleted
I20260925 05:13:31.232656 19477 ts_tablet_manager.cc:1939] T 875b5d0df5c44321b9d5bf81d706fae7 P 10a751c234ae46d2923344de5d229a5b: Deleting consensus metadata
I20260925 05:13:31.233150 18976 catalog_manager.cc:4841] TS 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): tablet 875b5d0df5c44321b9d5bf81d706fae7 (table test-table [id=b559ac7a33aa4fbab7a423b02e3d8caf]) successfully deleted
W20260925 05:13:31.235087 18976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:37449 (error 108)
I20260925 05:13:31.235652 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:31.235514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:37449 (local address 127.0.0.1:45242)
0925 05:13:31.235564 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:31.235609 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:37449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 05:13:31.235867 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 19183
W20260925 05:13:31.240437 18976 catalog_manager.cc:4584] TS 9a054633e36c468d83130306818630a1 (127.3.6.129:37449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 875b5d0df5c44321b9d5bf81d706fae7: Network error: Client connection negotiation failed: client connection to 127.3.6.129:37449: connect: Connection refused (error 111)
I20260925 05:13:31.241806 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 19319
I20260925 05:13:31.247440 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 18957
I20260925 05:13:51.253561 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32979
--webserver_interface=127.3.6.190
--webserver_port=37407
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:32979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:13:51.354931 19480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:51.355158 19480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:51.355181 19480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:51.356446 19480 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:13:51.356492 19480 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:13:51.356505 19480 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:13:51.356518 19480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:13:51.356580 19480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:51.356598 19480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:51.356623 19480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:13:51.356649 19480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:13:51.358201 19480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:32979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:51.358491 19480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:51.358778 19480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:51.364301 19488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:51.364312 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:51.364480 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:51.364535 19480 server_base.cc:1034] running on GCE node
I20260925 05:13:51.364818 19480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:51.365170 19480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:51.366335 19480 hybrid_clock.cc:648] HybridClock initialized: now 1790313231366322 us; error 27 us; skew 500 ppm
Sep 25 05:13:51 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:51.369045 19480 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:13:51.369558 19480 webserver.cc:460] Webserver started at http://127.3.6.190:37407/ using document root <none> and password file <none>
I20260925 05:13:51.369822 19480 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:51.369894 19480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:51.371348 19480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:13:51.372184 19495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.372421 19480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:13:51.372545 19480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "9a8b78fa59d14e4586b2f6597e793c08"
format_stamp: "Formatted at 2026-09-25 05:13:09 on dist-test-slave-1lvv"
server_key: "6110eaec16ba8fcc59770bcfedc4860d"
server_key_iv: "a64c6891d2d5f82edc8161b8c6dad477"
server_key_version: "encryptionkey@0"
I20260925 05:13:51.372956 19480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:51.380378 19480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:51.382546 19480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:51.382725 19480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:51.387069 19480 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:32979
I20260925 05:13:51.387102 19547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:32979 every 8 connection(s)
I20260925 05:13:51.387521 19480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:13:51.389073 19548 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:13:51.389699 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Bootstrap starting.
I20260925 05:13:51.391674 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 19480
I20260925 05:13:51.392004 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:37449
--local_ip_for_outbound_sockets=127.3.6.129
--tserver_master_addrs=127.3.6.190:32979
--webserver_port=44953
--webserver_interface=127.3.6.129
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:13:51.398459 19548 log.cc:824] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:51.400951 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:13:51.401291 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Bootstrap complete.
I20260925 05:13:51.403602 19548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:51.403923 19548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8b78fa59d14e4586b2f6597e793c08, State: Initialized, Role: FOLLOWER
I20260925 05:13:51.404150 19548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:51.404253 19548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:13:51.404307 19548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:13:51.404364 19548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:13:51.405391 19548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:51.405519 19548 leader_election.cc:302] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a8b78fa59d14e4586b2f6597e793c08; no voters:
I20260925 05:13:51.405735 19548 leader_election.cc:288] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:13:51.405778 19552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:13:51.405910 19552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [term 3 LEADER]: Becoming Leader. State: Replica: 9a8b78fa59d14e4586b2f6597e793c08, State: Running, Role: LEADER
I20260925 05:13:51.406021 19548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:13:51.406042 19552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } }
I20260925 05:13:51.406319 19552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8b78fa59d14e4586b2f6597e793c08. Latest consensus state: current_term: 3 leader_uuid: "9a8b78fa59d14e4586b2f6597e793c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } } }
I20260925 05:13:51.406400 19552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: This master's current role is: LEADER
I20260925 05:13:51.406324 19553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9a8b78fa59d14e4586b2f6597e793c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8b78fa59d14e4586b2f6597e793c08" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 32979 } } }
I20260925 05:13:51.406572 19553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08 [sys.catalog]: This master's current role is: LEADER
I20260925 05:13:51.406610 19557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:13:51.407513 19557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:13:51.407630 19557 catalog_manager.cc:1259] Loaded cluster ID: 5236ea3a1e794dcd94d5d229895ad202
I20260925 05:13:51.407668 19557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:13:51.408820 19557 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:13:51.409018 19557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Loaded TSK: 0
I20260925 05:13:51.409071 19557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a8b78fa59d14e4586b2f6597e793c08: Loaded TSK: 1
F20260925 05:13:51.409325 19557 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1d00e679ad google::LogMessage::Fail() at ??:0
@ 0x7f1d00e6a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1d00e674de google::LogMessage::Flush() at ??:0
@ 0x7f1d00e68a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1d035d4104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1d035d1d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1d08dc61cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1d08dc648e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1d08dc6594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1d08d8cc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1d08d96ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1d02d350d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1d02d2a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1d03b8e6db start_thread at ??:0
@ 0x7f1cff96561f clone at ??:0
W20260925 05:13:51.504511 19550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:51.504817 19550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:51.504859 19550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:51.506579 19550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:51.506651 19550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:51.506693 19550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:13:51.508971 19550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:37449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=44953
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:51.509272 19550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:51.509568 19550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:51.515096 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:51.515125 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:51.515090 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:51.515302 19550 server_base.cc:1034] running on GCE node
I20260925 05:13:51.515519 19550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:51.515826 19550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:51.517030 19550 hybrid_clock.cc:648] HybridClock initialized: now 1790313231517013 us; error 30 us; skew 500 ppm
Sep 25 05:13:51 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:51.520123 19550 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:13:51.520715 19550 webserver.cc:460] Webserver started at http://127.3.6.129:44953/ using document root <none> and password file <none>
I20260925 05:13:51.521020 19550 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:51.521100 19550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:51.522921 19550 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:13:51.523793 19599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.523980 19550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:13:51.524101 19550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "9a054633e36c468d83130306818630a1"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "693a43691d90e6209a6a8f3537167505"
server_key_iv: "8ab3ca97c88c662db90b08f0ba87c88f"
server_key_version: "encryptionkey@0"
I20260925 05:13:51.524466 19550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:51.540031 19550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:51.542423 19550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:51.542573 19550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:51.542847 19550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:51.543344 19606 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:13:51.545457 19550 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:13:51.545540 19550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 05:13:51.545589 19550 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:13:51.546278 19550 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:13:51.546340 19550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 05:13:51.546411 19606 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Bootstrap starting.
I20260925 05:13:51.554026 19550 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:37449
I20260925 05:13:51.554067 19713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:37449 every 8 connection(s)
I20260925 05:13:51.554546 19550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:13:51.557567 19606 log.cc:824] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:13:51.558169 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 19550
I20260925 05:13:51.558449 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:46609
--local_ip_for_outbound_sockets=127.3.6.130
--tserver_master_addrs=127.3.6.190:32979
--webserver_port=45387
--webserver_interface=127.3.6.130
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:13:51.560729 19606 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:13:51.561093 19606 tablet_bootstrap.cc:492] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Bootstrap complete.
I20260925 05:13:51.562371 19606 ts_tablet_manager.cc:1397] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260925 05:13:51.563992 19606 raft_consensus.cc:348] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:51.564345 19606 raft_consensus.cc:729] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a054633e36c468d83130306818630a1, State: Initialized, Role: FOLLOWER
I20260925 05:13:51.564518 19606 consensus_queue.cc:260] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:51.564834 19606 ts_tablet_manager.cc:1428] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260925 05:13:51.634593 19716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:13:51.555376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:41647)
0925 05:13:51.555559 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.555606 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:51.556255 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:51.634299 (+ 78044us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
W20260925 05:13:51.634897 19714 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:32979 (0 consecutive failures): Network error: Failed to ping master at 127.3.6.190:32979: Client connection negotiation failed: client connection to 127.3.6.190:32979: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:13:51.635241 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.635126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:58599)
0925 05:13:51.635177 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.635215 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 05:13:51.635612 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.635442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:54637)
0925 05:13:51.635518 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.635579 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 05:13:51.635866 19714 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:13:51.649817 19718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:51.650041 19718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:51.650066 19718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:51.651257 19718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:51.651302 19718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:51.651350 19718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:13:51.652961 19718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:46609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=45387
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:51.653214 19718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:51.653465 19718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:51.658824 19727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:51.658870 19725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:51.658838 19724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:51.659011 19718 server_base.cc:1034] running on GCE node
I20260925 05:13:51.659303 19718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:51.659571 19718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:51.660751 19718 hybrid_clock.cc:648] HybridClock initialized: now 1790313231660727 us; error 46 us; skew 500 ppm
Sep 25 05:13:51 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:51.663676 19718 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:13:51.664307 19718 webserver.cc:460] Webserver started at http://127.3.6.130:45387/ using document root <none> and password file <none>
I20260925 05:13:51.664572 19718 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:51.664623 19718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:51.666280 19718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:13:51.667025 19734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.667339 19718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:13:51.667428 19718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "10a751c234ae46d2923344de5d229a5b"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a0b7c7739a20e135f487ba5e7738db68"
server_key_iv: "d3674918fedab1ac625d764391fa8c3e"
server_key_version: "encryptionkey@0"
I20260925 05:13:51.667919 19718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:51.682760 19718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:51.684796 19718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:51.684975 19718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:51.685221 19718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:51.685544 19718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:13:51.685597 19718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.685642 19718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:13:51.685669 19718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.691946 19718 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:46609
I20260925 05:13:51.692004 19847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:46609 every 8 connection(s)
I20260925 05:13:51.692404 19718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:13:51.693637 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.693252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:44635)
0925 05:13:51.693414 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.693491 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260925 05:13:51.694082 19848 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:32979 (0 consecutive failures): Network error: Failed to ping master at 127.3.6.190:32979: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
I20260925 05:13:51.694372 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.694278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:42811)
0925 05:13:51.694324 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.694353 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 05:13:51.694610 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.694561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:60745)
0925 05:13:51.694580 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.694599 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 05:13:51.694713 19848 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:13:51.696228 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 19718
I20260925 05:13:51.696470 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:45153
--local_ip_for_outbound_sockets=127.3.6.131
--tserver_master_addrs=127.3.6.190:32979
--webserver_port=33251
--webserver_interface=127.3.6.131
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:13:51.778592 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:13:51.778864 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:13:51.778906 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:13:51.780108 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:13:51.780179 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:13:51.780224 19851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:13:51.781755 19851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:45153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=33251
--tserver_master_addrs=127.3.6.190:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:13:51.782035 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:13:51.782289 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:13:51.787561 19857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:13:51.787644 19859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:51.787791 19851 server_base.cc:1034] running on GCE node
W20260925 05:13:51.787791 19856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:13:51.788203 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:13:51.788492 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:13:51.789666 19851 hybrid_clock.cc:648] HybridClock initialized: now 1790313231789654 us; error 27 us; skew 500 ppm
Sep 25 05:13:51 dist-test-slave-1lvv krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:13:51.792340 19851 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:13:51.792846 19851 webserver.cc:460] Webserver started at http://127.3.6.131:33251/ using document root <none> and password file <none>
I20260925 05:13:51.793102 19851 fs_manager.cc:359] Metadata directory not provided
I20260925 05:13:51.793177 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:13:51.794755 19851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:13:51.795423 19866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.795653 19851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:13:51.795760 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "feb4987c963843b7a807ed3274da0b65"
format_stamp: "Formatted at 2026-09-25 05:13:10 on dist-test-slave-1lvv"
server_key: "a7c1161ad1451a0f759235454e3101da"
server_key_iv: "f3903d3f86cb6d51da6e5552b3ec2a2f"
server_key_version: "encryptionkey@0"
I20260925 05:13:51.796172 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:13:51.809028 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:13:51.811435 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:13:51.811651 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:13:51.811955 19851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:13:51.812320 19851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:13:51.812378 19851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.812412 19851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:13:51.812443 19851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:13:51.818708 19851 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:45153
I20260925 05:13:51.818750 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:45153 every 8 connection(s)
I20260925 05:13:51.819124 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:13:51.820257 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.819861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:40241)
0925 05:13:51.820035 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.820114 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
W20260925 05:13:51.820700 19980 heartbeater.cc:644] Failed to heartbeat to 127.3.6.190:32979 (0 consecutive failures): Network error: Failed to ping master at 127.3.6.190:32979: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
I20260925 05:13:51.820987 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.820892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:49943)
0925 05:13:51.820942 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.820970 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:13:51.821274 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:51.821200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:58821)
0925 05:13:51.821238 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:51.821260 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 05:13:51.821388 19980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:13:51.821931 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 19851
I20260925 05:13:52.636559 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:52.636291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:55175)
0925 05:13:52.636437 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:52.636484 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260925 05:13:52.695384 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:52.695134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:42589)
0925 05:13:52.695283 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:52.695326 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260925 05:13:52.821923 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:52.821698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:47529)
0925 05:13:52.821832 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:52.821872 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260925 05:13:53.415294 19987 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:13:53.415436 19987 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:53.415918 19987 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
Sep 25 05:13:53 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.130@KRBTEST.COM
Sep 25 05:13:53 dist-test-slave-1lvv krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.131@KRBTEST.COM
I20260925 05:13:53.429474 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:53.416142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:46609 (local address 127.3.6.129:32887)
0925 05:13:53.416290 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:53.416302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:53.416375 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:53.417385 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:53.417389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:53.417906 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:53.418229 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:53.418238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.420068 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.420069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:53.421401 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:53.421404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.422266 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.422267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:53.422452 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:53.423099 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:53.423104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:53.425037 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:53.427431 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:53.427442 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:53.427445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:53.427813 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:53.428146 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:53.428147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:53.428147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:53.428306 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:53.428823 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:53.428830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:53.428986 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 05:13:53.429108 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 05:13:53.429308 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":48,"thread_start_us":65,"threads_started":1}
I20260925 05:13:53.429481 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:53.416289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:32887 (local address 127.3.6.130:46609)
0925 05:13:53.416455 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:53.416468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:53.416494 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:13:53.417070 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:53.417074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:53.417136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:53.417303 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:53.418405 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.419917 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.421513 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.422156 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.422282 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:53.425161 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:53.425166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:53.425167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:53.425182 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:53.427332 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:53.427902 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:53.427905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:53.427906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:53.428048 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:53.428399 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:53.428400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:53.428400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:53.428745 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:53.428815 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:53.429187 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 05:13:53.429319 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260925 05:13:53.429605 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:53.416273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56687 (local address 127.3.6.131:45153)
0925 05:13:53.416460 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:13:53.416468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:13:53.416494 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:13:53.417070 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:13:53.417074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:13:53.417136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:13:53.417311 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:13:53.418405 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.419917 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.421512 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.422156 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.422282 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:53.425720 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:13:53.425726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:13:53.425726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:13:53.425741 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:13:53.427801 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:53.428276 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:53.428279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:53.428279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:53.428393 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:13:53.428605 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:13:53.428606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:13:53.428606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:13:53.428891 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:13:53.428975 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:13:53.429339 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 05:13:53.429496 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260925 05:13:53.429558 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:53.416164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:45153 (local address 127.3.6.129:56687)
0925 05:13:53.416290 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:53.416302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:13:53.416375 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:13:53.417385 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:13:53.417389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:13:53.417906 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:13:53.418229 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:53.418239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.420048 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.420050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:53.421401 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:13:53.421405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:13:53.422268 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:13:53.422268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:13:53.422452 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:13:53.423090 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:13:53.423101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:13:53.425604 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:13:53.427902 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:53.427903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:53.427904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:53.428199 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:53.428483 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:13:53.428484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:13:53.428484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:13:53.428547 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:13:53.429098 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:13:53.429099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:13:53.429154 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 05:13:53.429233 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:13:53.429380 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":146,"thread_start_us":60,"threads_started":1}
W20260925 05:13:53.430723 19603 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
W20260925 05:13:53.430814 19602 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:13:53.430841 19603 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a054633e36c468d83130306818630a1; no voters: 10a751c234ae46d2923344de5d229a5b, feb4987c963843b7a807ed3274da0b65
I20260925 05:13:53.430962 19987 raft_consensus.cc:2726] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:13:53.637351 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:53.637127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:55423)
0925 05:13:53.637220 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:53.637299 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 05:13:53.696309 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:53.696000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:37369)
0925 05:13:53.696156 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:53.696242 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260925 05:13:53.822752 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:53.822488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:38313)
0925 05:13:53.822638 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:53.822700 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260925 05:13:54.638222 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:54.637943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:44101)
0925 05:13:54.638115 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:54.638167 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260925 05:13:54.697042 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:54.696802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:36355)
0925 05:13:54.696948 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:54.696993 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260925 05:13:54.823463 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:54.823251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:45073)
0925 05:13:54.823379 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:54.823417 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260925 05:13:55.281076 19997 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:13:55.281165 19997 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:55.281380 19997 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
W20260925 05:13:55.281857 19602 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
W20260925 05:13:55.281939 19603 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:13:55.282050 19603 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a054633e36c468d83130306818630a1; no voters: 10a751c234ae46d2923344de5d229a5b, feb4987c963843b7a807ed3274da0b65
I20260925 05:13:55.282143 19997 raft_consensus.cc:2726] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:13:55.638913 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:55.638667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:42951)
0925 05:13:55.638815 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:55.638851 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260925 05:13:55.697784 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:55.697541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:46083)
0925 05:13:55.697681 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:55.697723 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260925 05:13:55.824299 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:55.824024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:50581)
0925 05:13:55.824192 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:55.824243 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260925 05:13:56.639663 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:56.639437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:34797)
0925 05:13:56.639574 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:56.639615 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260925 05:13:56.698673 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:56.698434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:43867)
0925 05:13:56.698573 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:56.698623 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":106,"threads_started":1}
I20260925 05:13:56.825095 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:56.824824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:38709)
0925 05:13:56.825001 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:56.825047 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260925 05:13:56.915495 20004 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:13:56.915596 20004 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:13:56.915918 20004 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
W20260925 05:13:56.916208 19603 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
W20260925 05:13:56.916350 19602 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:13:56.916410 19602 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a054633e36c468d83130306818630a1; no voters: 10a751c234ae46d2923344de5d229a5b, feb4987c963843b7a807ed3274da0b65
I20260925 05:13:56.916504 20004 raft_consensus.cc:2726] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:13:57.640396 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:57.640164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:32795)
0925 05:13:57.640303 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:57.640347 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260925 05:13:57.699395 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:57.699186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:46569)
0925 05:13:57.699310 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:57.699349 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260925 05:13:57.825923 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:57.825666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:37029)
0925 05:13:57.825830 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:57.825877 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260925 05:13:58.641166 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:58.640924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:40595)
0925 05:13:58.641072 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:58.641117 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260925 05:13:58.700241 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:58.699983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:48019)
0925 05:13:58.700136 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:58.700187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260925 05:13:58.826732 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:58.826476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:57425)
0925 05:13:58.826636 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:58.826683 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260925 05:13:59.641956 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:59.641718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:54517)
0925 05:13:59.641861 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:59.641905 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260925 05:13:59.701057 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:59.700826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:43871)
0925 05:13:59.700967 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:59.701010 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260925 05:13:59.827543 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:13:59.827295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:34337)
0925 05:13:59.827458 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:13:59.827494 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260925 05:14:00.642693 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:00.642446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:45781)
0925 05:14:00.642597 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:00.642635 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260925 05:14:00.701897 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:00.701607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:33949)
0925 05:14:00.701800 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:00.701850 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260925 05:14:00.828378 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:00.828135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:42809)
0925 05:14:00.828281 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:00.828329 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260925 05:14:00.851830 20017 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:00.851936 20017 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:14:00.852104 20017 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
W20260925 05:14:00.852556 19602 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
W20260925 05:14:00.852648 19603 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:14:00.852694 19603 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a054633e36c468d83130306818630a1; no voters: 10a751c234ae46d2923344de5d229a5b, feb4987c963843b7a807ed3274da0b65
I20260925 05:14:00.852794 20017 raft_consensus.cc:2726] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:14:01.643554 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:01.643278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:49121)
0925 05:14:01.643452 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:01.643498 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260925 05:14:01.702768 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:01.702499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:35017)
0925 05:14:01.702663 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:01.702714 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260925 05:14:01.829234 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:01.828962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:45919)
0925 05:14:01.829133 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:01.829183 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260925 05:14:02.644451 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:02.644207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:49731)
0925 05:14:02.644363 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:02.644403 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 05:14:02.661021 20022 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:02.661130 20022 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:14:02.661381 20022 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
W20260925 05:14:02.661800 19602 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
W20260925 05:14:02.661888 19603 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:14:02.661986 19603 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a054633e36c468d83130306818630a1; no voters: 10a751c234ae46d2923344de5d229a5b, feb4987c963843b7a807ed3274da0b65
I20260925 05:14:02.662082 20022 raft_consensus.cc:2726] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:14:02.703621 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:02.703355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:39791)
0925 05:14:02.703529 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:02.703576 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260925 05:14:02.830084 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:02.829825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:40677)
0925 05:14:02.829986 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:02.830036 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260925 05:14:03.645215 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:03.644984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:47981)
0925 05:14:03.645124 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:03.645165 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260925 05:14:03.704617 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:03.704207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:42375)
0925 05:14:03.704521 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:03.704570 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":236,"threads_started":1}
I20260925 05:14:03.831039 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:03.830712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:52739)
0925 05:14:03.830931 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:03.830988 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
I20260925 05:14:04.646083 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:04.645801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:46341)
0925 05:14:04.645986 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:04.646035 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260925 05:14:04.705358 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:04.705102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:57593)
0925 05:14:04.705261 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:04.705311 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260925 05:14:04.831807 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:04.831534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:38361)
0925 05:14:04.831670 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:04.831711 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260925 05:14:05.646929 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:05.646682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:47263)
0925 05:14:05.646838 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:05.646881 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260925 05:14:05.706130 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:05.705887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:35299)
0925 05:14:05.706036 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:05.706081 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260925 05:14:05.832605 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:05.832338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:45487)
0925 05:14:05.832519 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:05.832558 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260925 05:14:06.477361 20034 raft_consensus.cc:482] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:06.477502 20034 raft_consensus.cc:504] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a751c234ae46d2923344de5d229a5b" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 46609 } } peers { permanent_uuid: "feb4987c963843b7a807ed3274da0b65" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45153 } } peers { permanent_uuid: "9a054633e36c468d83130306818630a1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 37449 } }
I20260925 05:14:06.477725 20034 leader_election.cc:288] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609), feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153)
W20260925 05:14:06.478287 19602 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a751c234ae46d2923344de5d229a5b (127.3.6.130:46609): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
W20260925 05:14:06.478374 19603 leader_election.cc:341] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb4987c963843b7a807ed3274da0b65 (127.3.6.131:45153): Not found: Tablet not found: 875b5d0df5c44321b9d5bf81d706fae7
I20260925 05:14:06.478448 19603 leader_election.cc:302] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a054633e36c468d83130306818630a1; no voters: 10a751c234ae46d2923344de5d229a5b, feb4987c963843b7a807ed3274da0b65
I20260925 05:14:06.478545 20034 raft_consensus.cc:2726] T 875b5d0df5c44321b9d5bf81d706fae7 P 9a054633e36c468d83130306818630a1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:14:06.647980 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:06.647674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.129:59159)
0925 05:14:06.647875 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:06.647932 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":166,"threads_started":1}
I20260925 05:14:06.706966 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:06.706694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.130:56911)
0925 05:14:06.706869 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:06.706918 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260925 05:14:06.832307 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 19550
I20260925 05:14:06.834177 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:06.833924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:32979 (local address 127.3.6.131:52079)
0925 05:14:06.834060 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:06.834100 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.190:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260925 05:14:06.837931 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 19718
I20260925 05:14:06.842929 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 19851
2026-09-25T05:14:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58309 ms)
[----------] 30 tests from SecurityITest (238479 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:06 dist-test-slave-1lvv krb5kdc[20040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:06 dist-test-slave-1lvv krb5kdc[20040](info): set up 2 sockets
Sep 25 05:14:06 dist-test-slave-1lvv krb5kdc[20040](info): commencing operation
krb5kdc: starting...
W20260925 05:14:07.891660 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:07 dist-test-slave-1lvv krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:08.027707 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33831
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:08.111490 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:08.111781 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:08.111871 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:08.113139 20053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:08.113224 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:08.113250 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:08.113270 20053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:08.113303 20053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:08.114888 20053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:33831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:08.115195 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:08.115458 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:08.120654 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.120656 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.120627 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:08.120950 20053 server_base.cc:1034] running on GCE node
I20260925 05:14:08.121234 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:08.121557 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:08.122771 20053 hybrid_clock.cc:648] HybridClock initialized: now 1790313248122751 us; error 32 us; skew 500 ppm
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:08.125758 20053 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:08.126291 20053 webserver.cc:460] Webserver started at http://127.3.6.190:45935/ using document root <none> and password file <none>
I20260925 05:14:08.126590 20053 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:08.126670 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:08.126830 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:08.128070 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "81180ce924f64ea7a2fe4ba55547e755"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "2d4ec3ff4e523ac4f391c3e1a99ab7dc"
server_key_iv: "ea6d06cab90335ef09c118114c1364ed"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.128451 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "81180ce924f64ea7a2fe4ba55547e755"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "2d4ec3ff4e523ac4f391c3e1a99ab7dc"
server_key_iv: "ea6d06cab90335ef09c118114c1364ed"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.129954 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:08.130759 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.130955 20053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:08.131073 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "81180ce924f64ea7a2fe4ba55547e755"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "2d4ec3ff4e523ac4f391c3e1a99ab7dc"
server_key_iv: "ea6d06cab90335ef09c118114c1364ed"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.131155 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:08.140321 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:08.142603 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:08.142818 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:08.147111 20053 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:33831
I20260925 05:14:08.147146 20120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:33831 every 8 connection(s)
I20260925 05:14:08.147550 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:08.148523 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:08.150807 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755: Bootstrap starting.
I20260925 05:14:08.151376 20121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:08.151623 20121 log.cc:824] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:08.152396 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755: No bootstrap required, opened a new log
I20260925 05:14:08.153635 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20053
I20260925 05:14:08.153774 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:08.153982 3098 external_mini_cluster.cc:1428] Setting key 0764e9d5647810eed9bbe9cb83b09df6
I20260925 05:14:08.154078 20121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81180ce924f64ea7a2fe4ba55547e755" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33831 } }
I20260925 05:14:08.154291 20121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:08.154335 20121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81180ce924f64ea7a2fe4ba55547e755, State: Initialized, Role: FOLLOWER
I20260925 05:14:08.154500 20121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81180ce924f64ea7a2fe4ba55547e755" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33831 } }
I20260925 05:14:08.154600 20121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:08.154677 20121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:08.154755 20121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:08.155932 20121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81180ce924f64ea7a2fe4ba55547e755" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33831 } }
I20260925 05:14:08.156071 20121 leader_election.cc:302] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81180ce924f64ea7a2fe4ba55547e755; no voters:
I20260925 05:14:08.156227 20121 leader_election.cc:288] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:08.156299 20126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:08.156478 20126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [term 1 LEADER]: Becoming Leader. State: Replica: 81180ce924f64ea7a2fe4ba55547e755, State: Running, Role: LEADER
I20260925 05:14:08.156497 20121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:08.156577 20126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81180ce924f64ea7a2fe4ba55547e755" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33831 } }
I20260925 05:14:08.157028 20126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81180ce924f64ea7a2fe4ba55547e755. Latest consensus state: current_term: 1 leader_uuid: "81180ce924f64ea7a2fe4ba55547e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81180ce924f64ea7a2fe4ba55547e755" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33831 } } }
I20260925 05:14:08.157037 20127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81180ce924f64ea7a2fe4ba55547e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81180ce924f64ea7a2fe4ba55547e755" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33831 } } }
I20260925 05:14:08.157208 20127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:08.157202 20126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:08.157531 20135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:08.158896 20135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:08.160699 20135 catalog_manager.cc:1347] Generated new cluster ID: 3822ec518c1745d8a8c94ab38695a65d
I20260925 05:14:08.160766 20135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:08.166962 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.154667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.3.6.190:33831)
0925 05:14:08.154889 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.154900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.154936 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:14:08.155064 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.155072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.155143 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.155316 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:08.156023 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.157510 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.158561 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.159171 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.159285 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.162497 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:08.162502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:08.162502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:08.162517 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:08.164698 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.165120 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.165123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.165123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.165263 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.165476 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.165477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.165478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.165835 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:08.166494 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.166574 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.166733 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260925 05:14:08.170252 20135 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:08.170794 20135 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:08.177289 20135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755: Generated new TSK 0
I20260925 05:14:08.177505 20135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:14:08.211112 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33831
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:08.299619 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:08.299932 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:08.299963 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:08.301210 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:08.301256 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:08.301282 20147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:14:08.302924 20147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:08.303201 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:08.303460 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:08.308671 20153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.308700 20155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.308728 20152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:08.308956 20147 server_base.cc:1034] running on GCE node
I20260925 05:14:08.309207 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:08.309434 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:08.310595 20147 hybrid_clock.cc:648] HybridClock initialized: now 1790313248310587 us; error 33 us; skew 500 ppm
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:08.313478 20147 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:14:08.314020 20147 webserver.cc:460] Webserver started at http://127.3.6.129:45421/ using document root <none> and password file <none>
I20260925 05:14:08.314270 20147 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:08.314320 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:08.314500 20147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:08.315623 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "0f27339bfade40e08c16f301327e49c1"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "d85d891b3c3849b1c36077449d5d6c0c"
server_key_iv: "0932356596365f21db64748fb9cae8fd"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.316082 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "0f27339bfade40e08c16f301327e49c1"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "d85d891b3c3849b1c36077449d5d6c0c"
server_key_iv: "0932356596365f21db64748fb9cae8fd"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.317589 20147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:08.318369 20162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.318557 20147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:08.318681 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "0f27339bfade40e08c16f301327e49c1"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "d85d891b3c3849b1c36077449d5d6c0c"
server_key_iv: "0932356596365f21db64748fb9cae8fd"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.318756 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:08.335960 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:08.338532 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:08.338719 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:08.338986 20147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:08.339334 20147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:08.339389 20147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.339433 20147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:08.339476 20147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.345662 20147 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:40045
I20260925 05:14:08.345721 20275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:40045 every 8 connection(s)
I20260925 05:14:08.346073 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:14:08.347420 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20147
I20260925 05:14:08.347544 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:14:08.347749 3098 external_mini_cluster.cc:1428] Setting key f277a3311612639be94a5d6eb7774626
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:08.357388 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.346852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:53589 (local address 127.3.6.190:33831)
0925 05:14:08.346927 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.346929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.347169 (+ 240us) server_negotiation.cc:407] Connection header received
0925 05:14:08.348037 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.348039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.348056 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.348114 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:08.349508 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.350007 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.351176 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.351510 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.351591 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.353966 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:08.353969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:08.353969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:08.353978 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:08.355696 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.356157 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.356158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.356158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.356223 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.356608 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.356609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.356610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.356771 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:08.356859 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.357184 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.357274 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
I20260925 05:14:08.357560 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.346843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33831 (local address 127.3.6.129:53589)
0925 05:14:08.347031 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.347077 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.347863 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.348211 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.348216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.349094 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:08.349397 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.349406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.350099 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.350101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.351072 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.351075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.351600 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.351601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.351705 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.352362 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:08.352370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:08.353857 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:08.355826 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:08.355834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:08.355836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:08.356072 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:08.356331 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:08.356332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:08.356332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:08.356535 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:08.356851 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:08.356857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:08.356992 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.357084 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.357303 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260925 05:14:08.358179 20276 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33831
I20260925 05:14:08.358330 20276 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:08.358528 20276 heartbeater.cc:511] Master 127.3.6.190:33831 requested a full tablet report, sending...
I20260925 05:14:08.359058 20085 ts_manager.cc:194] Registered new tserver with Master: 0f27339bfade40e08c16f301327e49c1 (127.3.6.129:40045)
I20260925 05:14:08.359833 20085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:53589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:14:08.378793 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33831
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:08.462920 20281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:08.463331 20281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:08.463407 20281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:08.464983 20281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:08.465032 20281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:08.465088 20281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:14:08.466614 20281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:08.466882 20281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:08.467144 20281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:08.472594 20287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.472601 20286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:08.472744 20281 server_base.cc:1034] running on GCE node
W20260925 05:14:08.472821 20289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:08.473042 20281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:08.473299 20281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:08.474440 20281 hybrid_clock.cc:648] HybridClock initialized: now 1790313248474427 us; error 29 us; skew 500 ppm
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:08.477065 20281 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:14:08.477519 20281 webserver.cc:460] Webserver started at http://127.3.6.130:46193/ using document root <none> and password file <none>
I20260925 05:14:08.477764 20281 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:08.477810 20281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:08.477982 20281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:08.479086 20281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "1145ac4780254313b13a473328bcff02"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "d7ade35570c16ea69c166e2f8fb2efaa"
server_key_iv: "0917015d9e3ccf4308db73f9f8c6c94f"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.479458 20281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "1145ac4780254313b13a473328bcff02"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "d7ade35570c16ea69c166e2f8fb2efaa"
server_key_iv: "0917015d9e3ccf4308db73f9f8c6c94f"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.480935 20281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:08.481750 20296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.481956 20281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:08.482043 20281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "1145ac4780254313b13a473328bcff02"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "d7ade35570c16ea69c166e2f8fb2efaa"
server_key_iv: "0917015d9e3ccf4308db73f9f8c6c94f"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.482100 20281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:08.495600 20281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:08.497727 20281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:08.497869 20281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:08.498075 20281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:08.498426 20281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:08.498462 20281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.498486 20281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:08.498536 20281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.504670 20281 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:42325
I20260925 05:14:08.504760 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:42325 every 8 connection(s)
I20260925 05:14:08.505125 20281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:08.514803 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.505883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:50339 (local address 127.3.6.190:33831)
0925 05:14:08.505950 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.505952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.506200 (+ 248us) server_negotiation.cc:407] Connection header received
0925 05:14:08.506766 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.506767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.506781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.506833 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:08.507904 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.508426 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.509518 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.509731 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.509823 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.512153 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:08.512154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:08.512154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:08.512162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:08.513488 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.513878 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.513879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.513879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.513933 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.514179 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.514180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.514181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.514327 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:08.514411 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.514635 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.514730 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:14:08.514966 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.505876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33831 (local address 127.3.6.130:50339)
0925 05:14:08.506038 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.506095 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.506643 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.506912 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.506917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.507541 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:08.507742 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.507747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.508552 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.508553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.509399 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.509401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.509834 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.509836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.509969 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.510569 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:08.510579 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:08.512051 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:08.513569 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:08.513576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:08.513578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:08.513807 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:08.513991 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:08.513991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:08.513992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:08.514097 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:08.514386 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:08.514390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:08.514519 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.514604 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.514739 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260925 05:14:08.515231 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20281
I20260925 05:14:08.515352 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:14:08.515475 20410 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33831
I20260925 05:14:08.515568 3098 external_mini_cluster.cc:1428] Setting key fd87c97f5aeb448cb63c4405a598c580
I20260925 05:14:08.515604 20410 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:08.515838 20410 heartbeater.cc:511] Master 127.3.6.190:33831 requested a full tablet report, sending...
I20260925 05:14:08.516249 20085 ts_manager.cc:194] Registered new tserver with Master: 1145ac4780254313b13a473328bcff02 (127.3.6.130:42325)
I20260925 05:14:08.516767 20085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:50339
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:14:08.543915 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33831
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:08.628679 20415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:08.628942 20415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:08.628983 20415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:08.630183 20415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:08.630259 20415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:08.630304 20415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:14:08.632071 20415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:08.632362 20415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:08.632620 20415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:14:08.638296 20415 server_base.cc:1034] running on GCE node
W20260925 05:14:08.638238 20420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.638221 20423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.638257 20421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:08.638813 20415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:08.639081 20415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:08.640219 20415 hybrid_clock.cc:648] HybridClock initialized: now 1790313248640205 us; error 34 us; skew 500 ppm
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:08.643057 20415 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:14:08.643577 20415 webserver.cc:460] Webserver started at http://127.3.6.131:35177/ using document root <none> and password file <none>
I20260925 05:14:08.643891 20415 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:08.643945 20415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:08.644173 20415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:08.645386 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "dc6cfeb032f049698421c47405fbd8c6"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "7a9b4977c3d9251861cae03999b056a0"
server_key_iv: "a67ab211e3800d01275ea3e2bd00f24c"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.645787 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "dc6cfeb032f049698421c47405fbd8c6"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "7a9b4977c3d9251861cae03999b056a0"
server_key_iv: "a67ab211e3800d01275ea3e2bd00f24c"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.647322 20415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:08.648253 20430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.648468 20415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:14:08.648568 20415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "dc6cfeb032f049698421c47405fbd8c6"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "7a9b4977c3d9251861cae03999b056a0"
server_key_iv: "a67ab211e3800d01275ea3e2bd00f24c"
server_key_version: "encryptionkey@0"
I20260925 05:14:08.648648 20415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:08.673157 20415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:08.675700 20415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:08.675937 20415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:08.676223 20415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:08.676587 20415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:08.676623 20415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.676645 20415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:08.676694 20415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:08.683043 20415 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:42117
I20260925 05:14:08.683089 20543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:42117 every 8 connection(s)
I20260925 05:14:08.683442 20415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:14:08 dist-test-slave-1lvv krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:08.690189 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20415
I20260925 05:14:08.690315 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:14:08.690517 3098 external_mini_cluster.cc:1428] Setting key 50b1635de9f30f324be0ca13b39a7c8a
I20260925 05:14:08.693688 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.684229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:50011 (local address 127.3.6.190:33831)
0925 05:14:08.684348 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.684351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.684562 (+ 211us) server_negotiation.cc:407] Connection header received
0925 05:14:08.685211 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.685212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.685225 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.685285 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:08.686346 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.686859 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.687975 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.688176 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.688266 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.690711 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:08.690713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:08.690713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:08.690727 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:08.692334 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.692824 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.692825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.692826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.692885 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.693128 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.693128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.693128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.693238 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:08.693288 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.693550 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.693616 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:14:08.693991 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.684197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33831 (local address 127.3.6.131:50011)
0925 05:14:08.684416 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.684463 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.685090 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.685363 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.685367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.685985 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:08.686246 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.686252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.686969 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.686971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.687880 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.687883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.688278 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.688279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.688411 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.688957 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:08.688965 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:08.690542 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:08.692442 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:08.692450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:08.692453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:08.692724 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:08.692949 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:08.692950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:08.692950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:08.693067 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:08.693297 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:08.693301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:08.693432 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.693526 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.693665 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
I20260925 05:14:08.694550 20544 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33831
I20260925 05:14:08.694643 20544 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:08.694808 20544 heartbeater.cc:511] Master 127.3.6.190:33831 requested a full tablet report, sending...
I20260925 05:14:08.695214 20085 ts_manager.cc:194] Registered new tserver with Master: dc6cfeb032f049698421c47405fbd8c6 (127.3.6.131:42117)
I20260925 05:14:08.695618 20085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:50011
I20260925 05:14:08.702979 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:14:08.711402 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.704593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.3.6.190:33831)
0925 05:14:08.704678 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.704681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.704765 (+ 84us) server_negotiation.cc:407] Connection header received
0925 05:14:08.704865 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.704866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.704878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.704938 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:08.705678 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.706187 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.706946 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.707163 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.707253 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.708642 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:08.708643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:08.708644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:08.708652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:08.710112 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.710498 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.710499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.710500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.710561 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:08.710777 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:08.710778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:08.710778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:08.710948 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:08.711064 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.711200 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.711292 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:14:08.713428 20085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:14:08.715132 20085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:14:08.722290 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.717631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40045 (local address 127.0.0.1:41952)
0925 05:14:08.717753 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.717797 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.717864 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.718268 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.718272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.718291 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:08.718377 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.718379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.719836 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.719836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.720988 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.720992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.721944 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.721945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.722024 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.722039 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.722112 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.722176 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":61,"thread_start_us":67,"threads_started":1}
I20260925 05:14:08.722340 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.718456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.3.6.131:42117)
0925 05:14:08.718654 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.718659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.718668 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:14:08.718706 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.718707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.718726 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.718819 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:08.718949 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.719656 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.721370 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.721859 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.721954 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.722016 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.722190 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.722242 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260925 05:14:08.722479 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.717847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:42325 (local address 127.0.0.1:35842)
0925 05:14:08.718002 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.718014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.718074 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.718445 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.718446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.718448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:08.718481 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.718483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.719826 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.719828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.721136 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.721139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.722051 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.722052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.722118 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.722122 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.722229 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.722272 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":92,"thread_start_us":108,"threads_started":1}
I20260925 05:14:08.722280 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.718060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42117 (local address 127.0.0.1:51268)
0925 05:14:08.718274 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.718283 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.718328 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.718818 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.718819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.718821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:08.718879 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.718881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.719791 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.719792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.721284 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.721286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.722049 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.722050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.722111 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.722115 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.722167 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.722195 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":32,"thread_start_us":68,"threads_started":1}
I20260925 05:14:08.722687 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.717753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.3.6.129:40045)
0925 05:14:08.718062 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.718068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.718079 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:14:08.718135 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.718137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.718155 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.718245 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:08.718471 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.719684 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.721069 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.721841 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.721925 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.721993 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.722529 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.722585 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":100,"threads_started":1}
I20260925 05:14:08.722913 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.717984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.3.6.130:42325)
0925 05:14:08.718202 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.718205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.718235 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:14:08.718272 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.718274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.718290 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.718370 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:08.718559 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.719676 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.721217 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.721899 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.721957 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.722016 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.722340 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.722812 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260925 05:14:08.723351 20478 tablet_service.cc:1467] Processing CreateTablet for tablet d4d95e1cc6224a59ad1528009f7473da (DEFAULT_TABLE table=test-table [id=f0a1602da16f4a63bcfad50ad63b6364]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:08.723351 20210 tablet_service.cc:1467] Processing CreateTablet for tablet d4d95e1cc6224a59ad1528009f7473da (DEFAULT_TABLE table=test-table [id=f0a1602da16f4a63bcfad50ad63b6364]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:08.723516 20344 tablet_service.cc:1467] Processing CreateTablet for tablet d4d95e1cc6224a59ad1528009f7473da (DEFAULT_TABLE table=test-table [id=f0a1602da16f4a63bcfad50ad63b6364]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:08.723749 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d95e1cc6224a59ad1528009f7473da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:08.723749 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d95e1cc6224a59ad1528009f7473da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:08.723910 20344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d95e1cc6224a59ad1528009f7473da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:08.726576 20561 tablet_bootstrap.cc:492] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: Bootstrap starting.
I20260925 05:14:08.727298 20562 tablet_bootstrap.cc:492] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: Bootstrap starting.
I20260925 05:14:08.727494 20563 tablet_bootstrap.cc:492] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Bootstrap starting.
I20260925 05:14:08.727751 20561 tablet_bootstrap.cc:654] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:08.728171 20562 tablet_bootstrap.cc:654] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:08.728194 20563 tablet_bootstrap.cc:654] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:08.728572 20561 log.cc:824] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:08.728729 20563 log.cc:824] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:08.728816 20562 log.cc:824] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:08.729470 20561 tablet_bootstrap.cc:492] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: No bootstrap required, opened a new log
I20260925 05:14:08.729517 20563 tablet_bootstrap.cc:492] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: No bootstrap required, opened a new log
I20260925 05:14:08.729566 20561 ts_tablet_manager.cc:1397] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:14:08.729584 20562 tablet_bootstrap.cc:492] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: No bootstrap required, opened a new log
I20260925 05:14:08.729606 20563 ts_tablet_manager.cc:1397] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:14:08.729641 20562 ts_tablet_manager.cc:1397] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:14:08.731280 20563 raft_consensus.cc:348] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.731483 20563 raft_consensus.cc:374] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:08.731531 20563 raft_consensus.cc:729] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6cfeb032f049698421c47405fbd8c6, State: Initialized, Role: FOLLOWER
I20260925 05:14:08.731657 20563 consensus_queue.cc:260] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.731637 20562 raft_consensus.cc:348] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.731637 20561 raft_consensus.cc:348] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.731868 20561 raft_consensus.cc:374] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:08.731868 20562 raft_consensus.cc:374] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:08.731914 20561 raft_consensus.cc:729] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1145ac4780254313b13a473328bcff02, State: Initialized, Role: FOLLOWER
I20260925 05:14:08.731922 20562 raft_consensus.cc:729] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f27339bfade40e08c16f301327e49c1, State: Initialized, Role: FOLLOWER
I20260925 05:14:08.731998 20563 ts_tablet_manager.cc:1428] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:14:08.732043 20561 consensus_queue.cc:260] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.732153 20544 heartbeater.cc:503] Master 127.3.6.190:33831 was elected leader, sending a full tablet report...
I20260925 05:14:08.732337 20561 ts_tablet_manager.cc:1428] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260925 05:14:08.732453 20410 heartbeater.cc:503] Master 127.3.6.190:33831 was elected leader, sending a full tablet report...
I20260925 05:14:08.732084 20562 consensus_queue.cc:260] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.733008 20276 heartbeater.cc:503] Master 127.3.6.190:33831 was elected leader, sending a full tablet report...
I20260925 05:14:08.733019 20562 ts_tablet_manager.cc:1428] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260925 05:14:08.755697 20411 tablet.cc:2369] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:14:08.809649 20568 raft_consensus.cc:482] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:08.809798 20568 raft_consensus.cc:504] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.810223 20568 leader_election.cc:288] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f27339bfade40e08c16f301327e49c1 (127.3.6.129:40045), dc6cfeb032f049698421c47405fbd8c6 (127.3.6.131:42117)
I20260925 05:14:08.813431 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.810602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:42117 (local address 127.3.6.130:58727)
0925 05:14:08.810839 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.810851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.810910 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.811072 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.811073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.811075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:08.811140 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.811142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.811796 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.811797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.812591 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.812593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.813162 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.813163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.813233 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.813238 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.813289 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.813320 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":143,"threads_started":1}
I20260925 05:14:08.813442 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.810753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:58727 (local address 127.3.6.131:42117)
0925 05:14:08.810801 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.810803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.810952 (+ 149us) server_negotiation.cc:407] Connection header received
0925 05:14:08.810984 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.810985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.810998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.811050 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:08.811213 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.811663 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.812697 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.813078 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.813135 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.813143 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.813317 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.813368 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:14:08.813512 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.810604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:36165 (local address 127.3.6.129:40045)
0925 05:14:08.810658 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.810660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.810672 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:08.810703 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.810704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.810717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.810801 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:08.811004 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.811645 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.812696 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.813089 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.813182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.813192 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.813375 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.813420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:14:08.813534 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.810467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40045 (local address 127.3.6.130:36165)
0925 05:14:08.810550 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.810563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:08.810613 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:08.810817 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:08.810818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:08.810820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:08.810920 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.810922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.811739 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.811741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.812593 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:08.812596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.813199 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.813201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:08.813290 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.813294 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:08.813353 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:14:08.813390 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":44}
I20260925 05:14:08.813905 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d95e1cc6224a59ad1528009f7473da" candidate_uuid: "1145ac4780254313b13a473328bcff02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f27339bfade40e08c16f301327e49c1" is_pre_election: true
I20260925 05:14:08.813938 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d95e1cc6224a59ad1528009f7473da" candidate_uuid: "1145ac4780254313b13a473328bcff02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6cfeb032f049698421c47405fbd8c6" is_pre_election: true
I20260925 05:14:08.814085 20230 raft_consensus.cc:2454] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1145ac4780254313b13a473328bcff02 in term 0.
I20260925 05:14:08.814093 20498 raft_consensus.cc:2454] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1145ac4780254313b13a473328bcff02 in term 0.
I20260925 05:14:08.814332 20300 leader_election.cc:302] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1145ac4780254313b13a473328bcff02, dc6cfeb032f049698421c47405fbd8c6; no voters:
I20260925 05:14:08.814491 20568 raft_consensus.cc:2781] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:14:08.814570 20568 raft_consensus.cc:482] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:14:08.814590 20568 raft_consensus.cc:3037] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:08.815542 20568 raft_consensus.cc:504] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.815682 20568 leader_election.cc:288] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [CANDIDATE]: Term 1 election: Requested vote from peers 0f27339bfade40e08c16f301327e49c1 (127.3.6.129:40045), dc6cfeb032f049698421c47405fbd8c6 (127.3.6.131:42117)
I20260925 05:14:08.815846 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d95e1cc6224a59ad1528009f7473da" candidate_uuid: "1145ac4780254313b13a473328bcff02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6cfeb032f049698421c47405fbd8c6"
I20260925 05:14:08.815901 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d95e1cc6224a59ad1528009f7473da" candidate_uuid: "1145ac4780254313b13a473328bcff02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f27339bfade40e08c16f301327e49c1"
I20260925 05:14:08.815946 20498 raft_consensus.cc:3037] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:08.816010 20230 raft_consensus.cc:3037] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:08.816687 20498 raft_consensus.cc:2454] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1145ac4780254313b13a473328bcff02 in term 1.
I20260925 05:14:08.816838 20230 raft_consensus.cc:2454] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1145ac4780254313b13a473328bcff02 in term 1.
I20260925 05:14:08.816929 20300 leader_election.cc:302] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1145ac4780254313b13a473328bcff02, dc6cfeb032f049698421c47405fbd8c6; no voters:
I20260925 05:14:08.817023 20568 raft_consensus.cc:2781] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:08.817214 20568 raft_consensus.cc:686] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 1 LEADER]: Becoming Leader. State: Replica: 1145ac4780254313b13a473328bcff02, State: Running, Role: LEADER
I20260925 05:14:08.817348 20568 consensus_queue.cc:237] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } }
I20260925 05:14:08.818203 20084 catalog_manager.cc:5495] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1145ac4780254313b13a473328bcff02 (127.3.6.130). New cstate: current_term: 1 leader_uuid: "1145ac4780254313b13a473328bcff02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1145ac4780254313b13a473328bcff02" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 } health_report { overall_health: UNKNOWN } } }
I20260925 05:14:08.843294 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.840954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.3.6.130:42325)
0925 05:14:08.841027 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:08.841029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:08.841038 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:14:08.841076 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:08.841077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:08.841098 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:08.841161 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:08.841283 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.841771 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.842573 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:08.842762 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:08.842815 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:08.842950 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:08.843056 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:08.843120 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:08.843152 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 05:14:08.843172 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:14:08.845471 20498 raft_consensus.cc:1261] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 1 FOLLOWER]: Refusing update from remote peer 1145ac4780254313b13a473328bcff02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:08.845471 20230 raft_consensus.cc:1261] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 1 FOLLOWER]: Refusing update from remote peer 1145ac4780254313b13a473328bcff02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:08.845887 20568 consensus_queue.cc:1035] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc6cfeb032f049698421c47405fbd8c6" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:08.846050 20571 consensus_queue.cc:1035] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f27339bfade40e08c16f301327e49c1" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:14:08.846793 20277 tablet.cc:2369] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:14:08.848640 20574 mvcc.cc:205] Tried to move back new op lower bound from 7333123067267850240 to 7333123067156418560. Current Snapshot: MvccSnapshot[applied={T|T < 7333123067267850240}]
I20260925 05:14:08.849265 20576 mvcc.cc:205] Tried to move back new op lower bound from 7333123067267850240 to 7333123067156418560. Current Snapshot: MvccSnapshot[applied={T|T < 7333123067267850240}]
I20260925 05:14:08.851866 20085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:08.851994 20085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:08.852823 20085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81180ce924f64ea7a2fe4ba55547e755: Sending DeleteTablet for 3 replicas of tablet d4d95e1cc6224a59ad1528009f7473da
I20260925 05:14:08.853204 20344 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d95e1cc6224a59ad1528009f7473da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:08 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:35842
I20260925 05:14:08.853235 20210 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d95e1cc6224a59ad1528009f7473da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:08 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:41952
I20260925 05:14:08.853545 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d95e1cc6224a59ad1528009f7473da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:08 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:51268
I20260925 05:14:08.853741 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20147
I20260925 05:14:08.853789 20583 tablet_replica.cc:330] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1: stopping tablet replica
I20260925 05:14:08.853893 20583 raft_consensus.cc:2229] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:08.854008 20583 raft_consensus.cc:2258] T d4d95e1cc6224a59ad1528009f7473da P 0f27339bfade40e08c16f301327e49c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:08.854156 20585 tablet_replica.cc:330] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: stopping tablet replica
I20260925 05:14:08.854244 20585 raft_consensus.cc:2229] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:08.854337 20585 raft_consensus.cc:2258] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:08.854846 20585 ts_tablet_manager.cc:1905] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:08.857123 20585 ts_tablet_manager.cc:1918] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:08.857210 20585 log.cc:1196] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/d4d95e1cc6224a59ad1528009f7473da
I20260925 05:14:08.857522 20585 ts_tablet_manager.cc:1939] T d4d95e1cc6224a59ad1528009f7473da P dc6cfeb032f049698421c47405fbd8c6: Deleting consensus metadata
I20260925 05:14:08.858412 20072 catalog_manager.cc:4841] TS dc6cfeb032f049698421c47405fbd8c6 (127.3.6.131:42117): tablet d4d95e1cc6224a59ad1528009f7473da (table test-table [id=f0a1602da16f4a63bcfad50ad63b6364]) successfully deleted
I20260925 05:14:08.859913 20584 tablet_replica.cc:330] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: stopping tablet replica
I20260925 05:14:08.860054 20584 raft_consensus.cc:2229] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:14:08.860212 20584 raft_consensus.cc:2258] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:08.860697 20584 ts_tablet_manager.cc:1905] T d4d95e1cc6224a59ad1528009f7473da P 1145ac4780254313b13a473328bcff02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 05:14:08.862946 20072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:40045 (error 108)
I20260925 05:14:08.863197 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20281
I20260925 05:14:08.864461 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.864336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40045 (local address 127.0.0.1:41956)
0925 05:14:08.864381 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.864429 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 05:14:08.864596 20072 catalog_manager.cc:4584] TS 0f27339bfade40e08c16f301327e49c1 (127.3.6.129:40045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4d95e1cc6224a59ad1528009f7473da: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40045: connect: Connection refused (error 111)
I20260925 05:14:08.869144 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:08.869015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:42325 (local address 127.0.0.1:35860)
0925 05:14:08.869075 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:08.869105 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.130:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 05:14:08.869155 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20415
I20260925 05:14:08.873968 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20053
2026-09-25T05:14:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2030 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T05:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:08Z Disabled control of system clock
I20260925 05:14:08.904029 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:41377
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:41377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:14:08.987833 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:08.988080 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:08.988101 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:08.989336 20590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:08.989388 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:08.989401 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:08.989414 20590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:08.989434 20590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:08.990862 20590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:41377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:41377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:08.991153 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:08.991510 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:08.996940 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:08.997097 20590 server_base.cc:1034] running on GCE node
W20260925 05:14:08.996961 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:08.997186 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:08.997478 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:08.997790 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:08.998970 20590 hybrid_clock.cc:648] HybridClock initialized: now 1790313248998949 us; error 46 us; skew 500 ppm
I20260925 05:14:09.000366 20590 webserver.cc:460] Webserver started at http://127.3.6.190:35911/ using document root <none> and password file <none>
I20260925 05:14:09.000635 20590 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:09.000708 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:09.000867 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:09.001976 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "79b55840a6f7424499703872f6cce203"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "54777882dccbeead1899eee5883bb05a"
server_key_iv: "a176b5a73f9d1464b94c98b5a9b6c46b"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.002360 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "79b55840a6f7424499703872f6cce203"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "54777882dccbeead1899eee5883bb05a"
server_key_iv: "a176b5a73f9d1464b94c98b5a9b6c46b"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.003829 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 05:14:09.004648 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.004827 20590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:09.004933 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "79b55840a6f7424499703872f6cce203"
format_stamp: "Formatted at 2026-09-25 05:14:08 on dist-test-slave-1lvv"
server_key: "54777882dccbeead1899eee5883bb05a"
server_key_iv: "a176b5a73f9d1464b94c98b5a9b6c46b"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.005023 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:09.014055 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:09.014430 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:09.014601 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:09.018841 20590 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:41377
I20260925 05:14:09.018891 20656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:41377 every 8 connection(s)
I20260925 05:14:09.019284 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:09.020375 20657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:09.022677 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203: Bootstrap starting.
I20260925 05:14:09.023236 20657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:09.023488 20657 log.cc:824] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:09.024226 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203: No bootstrap required, opened a new log
I20260925 05:14:09.025857 20657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b55840a6f7424499703872f6cce203" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41377 } }
I20260925 05:14:09.026015 20657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:09.026041 20657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b55840a6f7424499703872f6cce203, State: Initialized, Role: FOLLOWER
I20260925 05:14:09.026199 20657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b55840a6f7424499703872f6cce203" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41377 } }
I20260925 05:14:09.026288 20657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:09.026314 20657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:09.026345 20657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:09.026888 20657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b55840a6f7424499703872f6cce203" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41377 } }
I20260925 05:14:09.026979 20657 leader_election.cc:302] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79b55840a6f7424499703872f6cce203; no voters:
I20260925 05:14:09.027145 20657 leader_election.cc:288] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:09.027232 20660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:09.027429 20660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [term 1 LEADER]: Becoming Leader. State: Replica: 79b55840a6f7424499703872f6cce203, State: Running, Role: LEADER
I20260925 05:14:09.027439 20657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:09.027551 20660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b55840a6f7424499703872f6cce203" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41377 } }
I20260925 05:14:09.028009 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b55840a6f7424499703872f6cce203. Latest consensus state: current_term: 1 leader_uuid: "79b55840a6f7424499703872f6cce203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b55840a6f7424499703872f6cce203" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41377 } } }
I20260925 05:14:09.028098 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79b55840a6f7424499703872f6cce203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b55840a6f7424499703872f6cce203" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 41377 } } }
I20260925 05:14:09.028112 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:09.028182 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:09.028435 20669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:09.028909 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20590
I20260925 05:14:09.029002 20669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:09.029052 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:09.029260 3098 external_mini_cluster.cc:1428] Setting key 7e5d52a8f6e1c48732b3c4cfa2119a70
I20260925 05:14:09.030791 20669 catalog_manager.cc:1347] Generated new cluster ID: dfccfbe2f54640fba37c571989a20155
I20260925 05:14:09.030867 20669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:09.036414 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.029749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.3.6.190:41377)
0925 05:14:09.029949 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.029959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.029992 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:14:09.030108 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.030115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.030186 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.030363 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:09.030552 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.032998 (+ 2446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.034093 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.034747 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.034888 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.035649 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:09.035653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:09.035653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:09.035898 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:09.035943 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:09.035958 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:09.036041 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.036110 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.036194 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260925 05:14:09.060360 20669 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:09.061141 20669 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:09.065809 20669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203: Generated new TSK 0
I20260925 05:14:09.066036 20669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:09.068555 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:41377
--builtin_ntp_servers=127.3.6.148:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:14:09.155894 20681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:09.156147 20681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:09.156186 20681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:09.157428 20681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:09.157502 20681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:09.157545 20681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:14:09.158980 20681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:41377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:09.159268 20681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:09.159610 20681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:09.164938 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:09.164932 20686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:09.164927 20687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:09.165401 20681 server_base.cc:1034] running on GCE node
I20260925 05:14:09.165617 20681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:09.165868 20681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:09.167047 20681 hybrid_clock.cc:648] HybridClock initialized: now 1790313249167037 us; error 27 us; skew 500 ppm
I20260925 05:14:09.168377 20681 webserver.cc:460] Webserver started at http://127.3.6.129:41511/ using document root <none> and password file <none>
I20260925 05:14:09.168633 20681 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:09.168704 20681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:09.168849 20681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:09.169956 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "6066b0f93172472a8feaba73740175fb"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "9517a526c3e29592e28f30f06ac9885b"
server_key_iv: "7b27dc0867f1491cde3b7cd4646cae85"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.170342 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "6066b0f93172472a8feaba73740175fb"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "9517a526c3e29592e28f30f06ac9885b"
server_key_iv: "7b27dc0867f1491cde3b7cd4646cae85"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.171814 20681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:14:09.172645 20695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.172822 20681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:09.172924 20681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "6066b0f93172472a8feaba73740175fb"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "9517a526c3e29592e28f30f06ac9885b"
server_key_iv: "7b27dc0867f1491cde3b7cd4646cae85"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.173023 20681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:09.181948 20681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:09.182298 20681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:09.182456 20681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:09.182703 20681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:09.183050 20681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:09.183105 20681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.183148 20681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:09.183182 20681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.189285 20681 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:36287
I20260925 05:14:09.189340 20808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:36287 every 8 connection(s)
I20260925 05:14:09.189728 20681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:14:09.194643 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20681
I20260925 05:14:09.194777 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:14:09.194993 3098 external_mini_cluster.cc:1428] Setting key bf3d8f0ce9c8bfb8c8a51ada40e3a271
I20260925 05:14:09.195523 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.190566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:41015 (local address 127.3.6.190:41377)
0925 05:14:09.190636 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.190638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.190911 (+ 273us) server_negotiation.cc:407] Connection header received
0925 05:14:09.191410 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.191411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.191424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.191480 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:09.192234 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.192740 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.193852 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.194047 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.194101 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.195146 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:09.195147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:09.195148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:09.195152 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:09.195168 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:09.195188 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:09.195255 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.195375 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.195430 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:14:09.195747 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.190579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:41377 (local address 127.3.6.129:41015)
0925 05:14:09.190783 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.190821 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.191320 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.191537 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.191540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.191549 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:09.192148 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.192153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.192860 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.192861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.193762 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.193765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.194127 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.194128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.194245 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.194950 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:09.194951 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:09.194998 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:09.195032 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:09.195264 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:09.195271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:09.195274 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.195371 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.195494 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260925 05:14:09.196149 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:41377
--builtin_ntp_servers=127.3.6.148:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:14:09.196415 20809 heartbeater.cc:347] Connected to a master server at 127.3.6.190:41377
I20260925 05:14:09.196520 20809 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:09.196738 20809 heartbeater.cc:511] Master 127.3.6.190:41377 requested a full tablet report, sending...
I20260925 05:14:09.197252 20621 ts_manager.cc:194] Registered new tserver with Master: 6066b0f93172472a8feaba73740175fb (127.3.6.129:36287)
I20260925 05:14:09.198153 20621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.129:41015
W20260925 05:14:09.283890 20812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:09.284117 20812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:09.284142 20812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:09.285302 20812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:09.285337 20812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:09.285363 20812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:14:09.286690 20812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:41377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:09.286897 20812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:09.287175 20812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:09.292817 20817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:09.292809 20820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:09.292811 20818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:09.293146 20812 server_base.cc:1034] running on GCE node
I20260925 05:14:09.293424 20812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:09.293735 20812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:09.294924 20812 hybrid_clock.cc:648] HybridClock initialized: now 1790313249294900 us; error 42 us; skew 500 ppm
I20260925 05:14:09.296546 20812 webserver.cc:460] Webserver started at http://127.3.6.130:46095/ using document root <none> and password file <none>
I20260925 05:14:09.296774 20812 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:09.296826 20812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:09.296929 20812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:09.298049 20812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "c2ab4c1aed67422590673df2242d5b24"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "e41be4996c43877376fd32a6451a10cf"
server_key_iv: "a479fbdf61a2277884a860a13c25c98c"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.298409 20812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "c2ab4c1aed67422590673df2242d5b24"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "e41be4996c43877376fd32a6451a10cf"
server_key_iv: "a479fbdf61a2277884a860a13c25c98c"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.300035 20812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:14:09.300794 20826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.300997 20812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:09.301085 20812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "c2ab4c1aed67422590673df2242d5b24"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "e41be4996c43877376fd32a6451a10cf"
server_key_iv: "a479fbdf61a2277884a860a13c25c98c"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.301142 20812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:09.325610 20812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:09.325933 20812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:09.326062 20812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:09.326301 20812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:09.326696 20812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:09.326735 20812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.326757 20812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:09.326807 20812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.333092 20812 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:44919
I20260925 05:14:09.333178 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:44919 every 8 connection(s)
I20260925 05:14:09.333667 20812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:14:09.339349 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.334545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:54323 (local address 127.3.6.190:41377)
0925 05:14:09.334631 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.334633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.334852 (+ 219us) server_negotiation.cc:407] Connection header received
0925 05:14:09.335436 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.335437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.335450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.335508 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:09.336317 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.336836 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.337887 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.338086 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.338168 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.339027 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:09.339029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:09.339029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:09.339034 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:09.339050 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:09.339054 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:09.339124 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.339219 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.339262 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:14:09.339614 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.334486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:41377 (local address 127.3.6.130:54323)
0925 05:14:09.334724 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.334763 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.335303 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.335593 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.335596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.335605 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:09.336212 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.336218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.336957 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.336963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.337778 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.337780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.338180 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.338181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.338322 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.338874 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:09.338875 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:09.338909 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:09.338933 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:09.339124 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:09.339131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:09.339134 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.339224 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.339330 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":103,"threads_started":1}
I20260925 05:14:09.340198 20940 heartbeater.cc:347] Connected to a master server at 127.3.6.190:41377
I20260925 05:14:09.340287 20940 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:09.340440 20940 heartbeater.cc:511] Master 127.3.6.190:41377 requested a full tablet report, sending...
I20260925 05:14:09.340796 20621 ts_manager.cc:194] Registered new tserver with Master: c2ab4c1aed67422590673df2242d5b24 (127.3.6.130:44919)
I20260925 05:14:09.341136 20621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.130:54323
I20260925 05:14:09.342473 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20812
I20260925 05:14:09.342610 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:14:09.342809 3098 external_mini_cluster.cc:1428] Setting key ce31ceb34669ad595cd7188c6f303ae5
I20260925 05:14:09.343973 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:41377
--builtin_ntp_servers=127.3.6.148:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:14:09.428468 20943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:09.428784 20943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:09.428812 20943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:09.430156 20943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:09.430207 20943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:09.430234 20943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:14:09.431628 20943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:41377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:09.431936 20943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:09.432229 20943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:09.437389 20951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:09.437361 20949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:09.437381 20948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:09.437575 20943 server_base.cc:1034] running on GCE node
I20260925 05:14:09.437880 20943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:09.438174 20943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:09.439339 20943 hybrid_clock.cc:648] HybridClock initialized: now 1790313249439326 us; error 44 us; skew 500 ppm
I20260925 05:14:09.440830 20943 webserver.cc:460] Webserver started at http://127.3.6.131:43373/ using document root <none> and password file <none>
I20260925 05:14:09.441108 20943 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:09.441184 20943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:09.441340 20943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:09.442480 20943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "df77fc1ce87a4c1ab872c9961f659f70"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "33112203cd9f25601b4a377449f1b175"
server_key_iv: "7464ce23010dbb8b27970c8810173232"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.442886 20943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "df77fc1ce87a4c1ab872c9961f659f70"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "33112203cd9f25601b4a377449f1b175"
server_key_iv: "7464ce23010dbb8b27970c8810173232"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.444402 20943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:09.445230 20957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.445446 20943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:09.445559 20943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "df77fc1ce87a4c1ab872c9961f659f70"
format_stamp: "Formatted at 2026-09-25 05:14:09 on dist-test-slave-1lvv"
server_key: "33112203cd9f25601b4a377449f1b175"
server_key_iv: "7464ce23010dbb8b27970c8810173232"
server_key_version: "encryptionkey@0"
I20260925 05:14:09.445645 20943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:09.463685 20943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:09.464093 20943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:09.464270 20943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:09.464531 20943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:09.464931 20943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:09.464994 20943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.465031 20943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:09.465076 20943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:09.471189 20943 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:39821
I20260925 05:14:09.471235 21070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:39821 every 8 connection(s)
I20260925 05:14:09.471731 20943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:14:09.477835 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.472927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:58231 (local address 127.3.6.190:41377)
0925 05:14:09.473041 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.473043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.473237 (+ 194us) server_negotiation.cc:407] Connection header received
0925 05:14:09.473800 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.473801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.473815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.473867 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:09.474623 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.475200 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.476340 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.476569 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.476640 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.477606 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:09.477607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:09.477607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:09.477612 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:09.477625 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:09.477629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:09.477672 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.477739 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.477768 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 05:14:09.478050 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.472801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:41377 (local address 127.3.6.131:58231)
0925 05:14:09.473107 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.473145 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.473679 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.473957 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.473961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.473970 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:09.474518 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.474523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.475292 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.475293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.476190 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.476193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.476657 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.476658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.476784 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.477452 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:09.477452 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:09.477486 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:09.477515 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:09.477673 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:09.477675 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:09.477678 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.477728 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.477818 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":111,"threads_started":1}
I20260925 05:14:09.478629 21071 heartbeater.cc:347] Connected to a master server at 127.3.6.190:41377
I20260925 05:14:09.478758 21071 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:09.478992 21071 heartbeater.cc:511] Master 127.3.6.190:41377 requested a full tablet report, sending...
I20260925 05:14:09.479444 20621 ts_manager.cc:194] Registered new tserver with Master: df77fc1ce87a4c1ab872c9961f659f70 (127.3.6.131:39821)
I20260925 05:14:09.479913 20621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.131:58231
I20260925 05:14:09.480031 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 20943
I20260925 05:14:09.480140 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:14:09.480347 3098 external_mini_cluster.cc:1428] Setting key 193b0829e7b50f4a31601d5e63db9b5f
I20260925 05:14:09.481280 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:14:09.485927 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.482790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.3.6.190:41377)
0925 05:14:09.482876 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.482879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.482959 (+ 80us) server_negotiation.cc:407] Connection header received
0925 05:14:09.483039 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.483040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.483051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.483100 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:09.483284 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.483834 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.484635 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.484844 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.484935 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.485675 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:09.485677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:09.485677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:09.485682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:09.485695 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:09.485699 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:09.485752 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.485806 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.485829 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:14:09.487661 20621 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:14:09.489513 20621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:14:09.495999 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.491687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:39821 (local address 127.0.0.1:53992)
0925 05:14:09.491998 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.492009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.492069 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.492420 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.492424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.492426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:09.492508 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.492510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.493473 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.493475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.494654 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.494657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.495629 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.495630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.495724 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.495742 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.495836 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.495902 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260925 05:14:09.496083 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.492160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.3.6.130:44919)
0925 05:14:09.492360 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.492364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.492373 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:14:09.492410 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.492412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.492426 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.492512 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:09.492699 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.493855 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.495068 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.495672 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.495749 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.495873 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.495964 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.496004 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260925 05:14:09.496169 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.491674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36287 (local address 127.0.0.1:51436)
0925 05:14:09.491902 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.491954 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.492024 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.492522 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.492522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.492524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:09.492577 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.492579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.493950 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.493951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.494991 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.494994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.495948 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.495949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.496012 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.496016 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.496047 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.496084 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260925 05:14:09.496371 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.491815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.3.6.131:39821)
0925 05:14:09.492204 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.492209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.492222 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:14:09.492269 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.492271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.492287 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.492599 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:09.492668 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.493369 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.494751 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.495543 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.495617 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.495720 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.496042 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.496091 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":99,"threads_started":1}
I20260925 05:14:09.496438 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.491914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:44919 (local address 127.0.0.1:56212)
0925 05:14:09.492146 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.492154 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.492198 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.492584 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.492585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.492587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:09.492622 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.492623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.493995 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.493996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.494963 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.494966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.495815 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.495816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.495879 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.495884 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.495948 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.495990 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260925 05:14:09.497056 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.491792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.3.6.129:36287)
0925 05:14:09.492329 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.492334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.492346 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:09.492392 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.492394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.492410 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.492656 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:09.492727 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.493856 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.495097 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.495852 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.496784 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.496892 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.496917 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.496959 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":98,"threads_started":1}
I20260925 05:14:09.497251 20874 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26f05fd89f4aeab149f12d19f03e31 (DEFAULT_TABLE table=test-table [id=abc2831e226e495da15c919320a489a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:09.497248 21005 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26f05fd89f4aeab149f12d19f03e31 (DEFAULT_TABLE table=test-table [id=abc2831e226e495da15c919320a489a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:09.497644 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a26f05fd89f4aeab149f12d19f03e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:09.497644 21005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a26f05fd89f4aeab149f12d19f03e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:09.497735 20743 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26f05fd89f4aeab149f12d19f03e31 (DEFAULT_TABLE table=test-table [id=abc2831e226e495da15c919320a489a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:09.498000 20743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a26f05fd89f4aeab149f12d19f03e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:09.500852 21085 tablet_bootstrap.cc:492] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb: Bootstrap starting.
I20260925 05:14:09.501296 21087 tablet_bootstrap.cc:492] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Bootstrap starting.
I20260925 05:14:09.501293 21086 tablet_bootstrap.cc:492] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Bootstrap starting.
I20260925 05:14:09.501648 21085 tablet_bootstrap.cc:654] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:09.501952 21085 log.cc:824] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:09.501987 21086 tablet_bootstrap.cc:654] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:09.501991 21087 tablet_bootstrap.cc:654] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:09.502303 21087 log.cc:824] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:09.502303 21086 log.cc:824] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:09.502766 21085 tablet_bootstrap.cc:492] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb: No bootstrap required, opened a new log
I20260925 05:14:09.502847 21085 ts_tablet_manager.cc:1397] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:14:09.503094 21087 tablet_bootstrap.cc:492] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: No bootstrap required, opened a new log
I20260925 05:14:09.503170 21087 ts_tablet_manager.cc:1397] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:14:09.503408 21086 tablet_bootstrap.cc:492] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: No bootstrap required, opened a new log
I20260925 05:14:09.503479 21086 ts_tablet_manager.cc:1397] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:14:09.505376 21085 raft_consensus.cc:348] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.505548 21085 raft_consensus.cc:374] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:09.505743 21085 raft_consensus.cc:729] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6066b0f93172472a8feaba73740175fb, State: Initialized, Role: FOLLOWER
I20260925 05:14:09.505919 21085 consensus_queue.cc:260] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.505980 21087 raft_consensus.cc:348] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.506157 21087 raft_consensus.cc:374] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:09.506211 21087 raft_consensus.cc:729] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ab4c1aed67422590673df2242d5b24, State: Initialized, Role: FOLLOWER
I20260925 05:14:09.506227 21085 ts_tablet_manager.cc:1428] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 05:14:09.506203 21086 raft_consensus.cc:348] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.506359 21086 raft_consensus.cc:374] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:09.506383 20809 heartbeater.cc:503] Master 127.3.6.190:41377 was elected leader, sending a full tablet report...
I20260925 05:14:09.506417 21086 raft_consensus.cc:729] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df77fc1ce87a4c1ab872c9961f659f70, State: Initialized, Role: FOLLOWER
I20260925 05:14:09.506537 21086 consensus_queue.cc:260] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.506820 21086 ts_tablet_manager.cc:1428] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 05:14:09.506778 21087 consensus_queue.cc:260] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.506872 21071 heartbeater.cc:503] Master 127.3.6.190:41377 was elected leader, sending a full tablet report...
I20260925 05:14:09.507128 21087 ts_tablet_manager.cc:1428] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260925 05:14:09.507148 20940 heartbeater.cc:503] Master 127.3.6.190:41377 was elected leader, sending a full tablet report...
I20260925 05:14:09.557420 21091 raft_consensus.cc:482] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:09.557569 21091 raft_consensus.cc:504] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.557912 21091 leader_election.cc:288] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ab4c1aed67422590673df2242d5b24 (127.3.6.130:44919), df77fc1ce87a4c1ab872c9961f659f70 (127.3.6.131:39821)
I20260925 05:14:09.561201 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.558265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:43051 (local address 127.3.6.130:44919)
0925 05:14:09.558335 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.558337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.558345 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:09.558373 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.558373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.558384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.558479 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:09.558708 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.559255 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.560393 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.560803 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.560865 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.560875 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.561054 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.561110 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:14:09.561233 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.558218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:60307 (local address 127.3.6.131:39821)
0925 05:14:09.558275 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.558277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.558600 (+ 323us) server_negotiation.cc:407] Connection header received
0925 05:14:09.558637 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.558638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.558649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.558710 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:09.558864 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.559353 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.560463 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.560826 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.560931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.560941 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.561122 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.561167 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 05:14:09.561223 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.558130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:44919 (local address 127.3.6.129:43051)
0925 05:14:09.558185 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.558197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.558244 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.558485 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.558487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.558489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:09.558564 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.558585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.559344 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.559345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.560287 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.560290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.560893 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.560895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.560970 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.560974 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.561030 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.561068 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 05:14:09.561242 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.558130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:39821 (local address 127.3.6.129:60307)
0925 05:14:09.558521 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.558531 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:09.558590 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:09.558723 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:09.558724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:09.558725 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:09.558794 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.558796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.559463 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.559464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.560360 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:09.560363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.560954 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.560955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:09.561037 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.561042 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:14:09.561102 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:14:09.561141 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":165,"thread_start_us":162,"threads_started":1}
I20260925 05:14:09.561667 20894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a26f05fd89f4aeab149f12d19f03e31" candidate_uuid: "6066b0f93172472a8feaba73740175fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ab4c1aed67422590673df2242d5b24" is_pre_election: true
I20260925 05:14:09.561856 20894 raft_consensus.cc:2454] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6066b0f93172472a8feaba73740175fb in term 0.
I20260925 05:14:09.561827 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a26f05fd89f4aeab149f12d19f03e31" candidate_uuid: "6066b0f93172472a8feaba73740175fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df77fc1ce87a4c1ab872c9961f659f70" is_pre_election: true
I20260925 05:14:09.561998 21025 raft_consensus.cc:2454] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6066b0f93172472a8feaba73740175fb in term 0.
I20260925 05:14:09.562189 20697 leader_election.cc:302] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6066b0f93172472a8feaba73740175fb, c2ab4c1aed67422590673df2242d5b24; no voters:
I20260925 05:14:09.562367 21091 raft_consensus.cc:2781] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:14:09.562431 21091 raft_consensus.cc:482] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:14:09.562457 21091 raft_consensus.cc:3037] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:09.563212 21091 raft_consensus.cc:504] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.563381 21091 leader_election.cc:288] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [CANDIDATE]: Term 1 election: Requested vote from peers c2ab4c1aed67422590673df2242d5b24 (127.3.6.130:44919), df77fc1ce87a4c1ab872c9961f659f70 (127.3.6.131:39821)
I20260925 05:14:09.563575 21025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a26f05fd89f4aeab149f12d19f03e31" candidate_uuid: "6066b0f93172472a8feaba73740175fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df77fc1ce87a4c1ab872c9961f659f70"
I20260925 05:14:09.563607 20894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a26f05fd89f4aeab149f12d19f03e31" candidate_uuid: "6066b0f93172472a8feaba73740175fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ab4c1aed67422590673df2242d5b24"
I20260925 05:14:09.563664 21025 raft_consensus.cc:3037] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:09.563699 20894 raft_consensus.cc:3037] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:09.564486 20894 raft_consensus.cc:2454] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6066b0f93172472a8feaba73740175fb in term 1.
I20260925 05:14:09.564509 21025 raft_consensus.cc:2454] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6066b0f93172472a8feaba73740175fb in term 1.
I20260925 05:14:09.564685 20697 leader_election.cc:302] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6066b0f93172472a8feaba73740175fb, c2ab4c1aed67422590673df2242d5b24; no voters:
I20260925 05:14:09.564800 21091 raft_consensus.cc:2781] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:09.564989 21091 raft_consensus.cc:686] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 1 LEADER]: Becoming Leader. State: Replica: 6066b0f93172472a8feaba73740175fb, State: Running, Role: LEADER
I20260925 05:14:09.565101 21091 consensus_queue.cc:237] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } }
I20260925 05:14:09.565984 20620 catalog_manager.cc:5495] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6066b0f93172472a8feaba73740175fb (127.3.6.129). New cstate: current_term: 1 leader_uuid: "6066b0f93172472a8feaba73740175fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6066b0f93172472a8feaba73740175fb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 } health_report { overall_health: UNKNOWN } } }
W20260925 05:14:09.584388 20941 tablet.cc:2369] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:14:09.589680 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.587067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.3.6.129:36287)
0925 05:14:09.587137 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:09.587139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:09.587167 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:14:09.587221 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:09.587222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:09.587243 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:09.587322 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:09.587492 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.588016 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.588829 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:09.589037 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:09.589118 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:09.589305 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:09.589418 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:09.589518 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:09.589543 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 05:14:09.589566 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:14:09.591593 21025 raft_consensus.cc:1261] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 1 FOLLOWER]: Refusing update from remote peer 6066b0f93172472a8feaba73740175fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:09.591602 20894 raft_consensus.cc:1261] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 1 FOLLOWER]: Refusing update from remote peer 6066b0f93172472a8feaba73740175fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:09.592010 21095 consensus_queue.cc:1035] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df77fc1ce87a4c1ab872c9961f659f70" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:09.592155 21091 consensus_queue.cc:1035] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ab4c1aed67422590673df2242d5b24" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 44919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:09.597255 20621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:09.597366 20621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:09.598170 20621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79b55840a6f7424499703872f6cce203: Sending DeleteTablet for 3 replicas of tablet 1a26f05fd89f4aeab149f12d19f03e31
I20260925 05:14:09.598528 20743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26f05fd89f4aeab149f12d19f03e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:09 UTC) from {username='slave'} at 127.0.0.1:51436
I20260925 05:14:09.598529 20874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26f05fd89f4aeab149f12d19f03e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:09 UTC) from {username='slave'} at 127.0.0.1:56212
I20260925 05:14:09.598537 21005 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a26f05fd89f4aeab149f12d19f03e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:09 UTC) from {username='slave'} at 127.0.0.1:53992
I20260925 05:14:09.598752 21108 tablet_replica.cc:330] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: stopping tablet replica
I20260925 05:14:09.598865 21108 raft_consensus.cc:2229] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:09.598977 21108 raft_consensus.cc:2258] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:09.599018 21109 tablet_replica.cc:330] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb: stopping tablet replica
I20260925 05:14:09.599103 21109 raft_consensus.cc:2229] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:14:09.599136 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20681
I20260925 05:14:09.599201 21109 raft_consensus.cc:2258] T 1a26f05fd89f4aeab149f12d19f03e31 P 6066b0f93172472a8feaba73740175fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:09.599434 21108 ts_tablet_manager.cc:1905] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:09.601168 21110 tablet_replica.cc:330] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: stopping tablet replica
I20260925 05:14:09.601285 21110 raft_consensus.cc:2229] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:09.601395 21110 raft_consensus.cc:2258] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:09.601656 21108 ts_tablet_manager.cc:1918] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:09.601756 21108 log.cc:1196] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/1a26f05fd89f4aeab149f12d19f03e31
I20260925 05:14:09.601833 21110 ts_tablet_manager.cc:1905] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:09.602037 21108 ts_tablet_manager.cc:1939] T 1a26f05fd89f4aeab149f12d19f03e31 P c2ab4c1aed67422590673df2242d5b24: Deleting consensus metadata
I20260925 05:14:09.602478 20606 catalog_manager.cc:4841] TS c2ab4c1aed67422590673df2242d5b24 (127.3.6.130:44919): tablet 1a26f05fd89f4aeab149f12d19f03e31 (table test-table [id=abc2831e226e495da15c919320a489a0]) successfully deleted
I20260925 05:14:09.603264 21110 ts_tablet_manager.cc:1918] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:09.603382 21110 log.cc:1196] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/1a26f05fd89f4aeab149f12d19f03e31
I20260925 05:14:09.603612 21110 ts_tablet_manager.cc:1939] T 1a26f05fd89f4aeab149f12d19f03e31 P df77fc1ce87a4c1ab872c9961f659f70: Deleting consensus metadata
I20260925 05:14:09.604100 20605 catalog_manager.cc:4841] TS df77fc1ce87a4c1ab872c9961f659f70 (127.3.6.131:39821): tablet 1a26f05fd89f4aeab149f12d19f03e31 (table test-table [id=abc2831e226e495da15c919320a489a0]) successfully deleted
W20260925 05:14:09.605612 20606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:36287 (error 108)
I20260925 05:14:09.605857 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20812
I20260925 05:14:09.606596 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:09.605996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36287 (local address 127.0.0.1:51440)
0925 05:14:09.606508 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:09.606555 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488}
W20260925 05:14:09.607510 20606 catalog_manager.cc:4584] TS 6066b0f93172472a8feaba73740175fb (127.3.6.129:36287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a26f05fd89f4aeab149f12d19f03e31: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36287: connect: Connection refused (error 111)
I20260925 05:14:09.611454 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20943
I20260925 05:14:09.616633 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 20590
2026-09-25T05:14:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (742 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:09 dist-test-slave-1lvv krb5kdc[21114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:09 dist-test-slave-1lvv krb5kdc[21114](info): set up 2 sockets
Sep 25 05:14:09 dist-test-slave-1lvv krb5kdc[21114](info): commencing operation
krb5kdc: starting...
W20260925 05:14:10.656028 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:10 dist-test-slave-1lvv krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:10.791225 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44579
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:10.875000 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:10.875291 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:10.875336 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:10.876662 21127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:10.876757 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:10.876782 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:10.876803 21127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:10.876860 21127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:10.878510 21127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:10.878854 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:10.879130 21127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:10.884353 21135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:10.884366 21133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:10.884455 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:10.884634 21127 server_base.cc:1034] running on GCE node
I20260925 05:14:10.884881 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:10.885154 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:10.886338 21127 hybrid_clock.cc:648] HybridClock initialized: now 1790313250886322 us; error 36 us; skew 500 ppm
Sep 25 05:14:10 dist-test-slave-1lvv krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:10.889293 21127 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:10.889814 21127 webserver.cc:460] Webserver started at http://127.3.6.190:38735/ using document root <none> and password file <none>
I20260925 05:14:10.890057 21127 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:10.890102 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:10.890202 21127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:10.891284 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "a113e2ed9c654c439bf915d47166883e"
format_stamp: "Formatted at 2026-09-25 05:14:10 on dist-test-slave-1lvv"
server_key: "b7c11eed2f4db50fd9c9ebbccd85e1f5"
server_key_iv: "0be6e8d0a7b7903d9db43c3c74b97d05"
server_key_version: "encryptionkey@0"
I20260925 05:14:10.891614 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "a113e2ed9c654c439bf915d47166883e"
format_stamp: "Formatted at 2026-09-25 05:14:10 on dist-test-slave-1lvv"
server_key: "b7c11eed2f4db50fd9c9ebbccd85e1f5"
server_key_iv: "0be6e8d0a7b7903d9db43c3c74b97d05"
server_key_version: "encryptionkey@0"
I20260925 05:14:10.893185 21127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:10.893983 21142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:10.894196 21127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:10.894301 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "a113e2ed9c654c439bf915d47166883e"
format_stamp: "Formatted at 2026-09-25 05:14:10 on dist-test-slave-1lvv"
server_key: "b7c11eed2f4db50fd9c9ebbccd85e1f5"
server_key_iv: "0be6e8d0a7b7903d9db43c3c74b97d05"
server_key_version: "encryptionkey@0"
I20260925 05:14:10.894371 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:10.917281 21127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:10.919683 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:10.919890 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:10.924072 21127 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44579
I20260925 05:14:10.924106 21194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44579 every 8 connection(s)
I20260925 05:14:10.924535 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:10.925537 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:10.927534 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 21127
I20260925 05:14:10.927662 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:10.927891 3098 external_mini_cluster.cc:1428] Setting key 9deb34c705679f25f3e3c196e7afcbdf
I20260925 05:14:10.928484 21195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e: Bootstrap starting.
I20260925 05:14:10.929189 21195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:10.929548 21195 log.cc:824] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:10.930397 21195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e: No bootstrap required, opened a new log
I20260925 05:14:10.932626 21195 raft_consensus.cc:348] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a113e2ed9c654c439bf915d47166883e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44579 } }
I20260925 05:14:10.932770 21195 raft_consensus.cc:374] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:10.932791 21195 raft_consensus.cc:729] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a113e2ed9c654c439bf915d47166883e, State: Initialized, Role: FOLLOWER
I20260925 05:14:10.932940 21195 consensus_queue.cc:260] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a113e2ed9c654c439bf915d47166883e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44579 } }
I20260925 05:14:10.933022 21195 raft_consensus.cc:388] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:10.933048 21195 raft_consensus.cc:482] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:10.933118 21195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:10.933868 21195 raft_consensus.cc:504] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a113e2ed9c654c439bf915d47166883e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44579 } }
I20260925 05:14:10.933995 21195 leader_election.cc:302] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a113e2ed9c654c439bf915d47166883e; no voters:
I20260925 05:14:10.934211 21195 leader_election.cc:288] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:10.934238 21200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:10.934366 21200 raft_consensus.cc:686] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [term 1 LEADER]: Becoming Leader. State: Replica: a113e2ed9c654c439bf915d47166883e, State: Running, Role: LEADER
I20260925 05:14:10.934451 21200 consensus_queue.cc:237] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a113e2ed9c654c439bf915d47166883e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44579 } }
I20260925 05:14:10.934490 21195 sys_catalog.cc:564] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:10.934772 21200 sys_catalog.cc:455] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a113e2ed9c654c439bf915d47166883e. Latest consensus state: current_term: 1 leader_uuid: "a113e2ed9c654c439bf915d47166883e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a113e2ed9c654c439bf915d47166883e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44579 } } }
I20260925 05:14:10.934846 21200 sys_catalog.cc:458] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:10.934790 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a113e2ed9c654c439bf915d47166883e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a113e2ed9c654c439bf915d47166883e" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44579 } } }
I20260925 05:14:10.935062 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e [sys.catalog]: This master's current role is: LEADER
Sep 25 05:14:10 dist-test-slave-1lvv krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:10.936012 21205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:10.936683 21205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:10.938745 21205 catalog_manager.cc:1347] Generated new cluster ID: ccfb57f48607441187d17b8225caa81e
I20260925 05:14:10.938805 21205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:10.940013 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:10.928526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.3.6.190:44579)
0925 05:14:10.928720 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:10.928729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:10.928756 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:14:10.929223 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:10.929231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:10.929304 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:10.929462 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:10.930294 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:10.931237 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:10.932465 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:10.932998 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:10.933146 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:10.936225 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:10.936229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:10.936230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:10.936243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:10.938257 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:10.938698 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:10.938701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:10.938701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:10.938811 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:10.939058 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:10.939059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:10.939059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:10.939311 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:10.939521 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:10.939623 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 05:14:10.939793 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260925 05:14:10.959195 21205 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:10.960074 21205 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:10.969519 21205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e: Generated new TSK 0
I20260925 05:14:10.969789 21205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:14:11.010857 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44579
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:11.097090 21221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:11.097424 21221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:11.097469 21221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:11.098901 21221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:11.098971 21221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:11.099042 21221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:14:11.100735 21221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:11.101040 21221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:11.101290 21221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:11.106726 21229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:11.106738 21227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:11.106719 21226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:11.107096 21221 server_base.cc:1034] running on GCE node
I20260925 05:14:11.107280 21221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:11.107532 21221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:11.108739 21221 hybrid_clock.cc:648] HybridClock initialized: now 1790313251108702 us; error 48 us; skew 500 ppm
Sep 25 05:14:11 dist-test-slave-1lvv krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:11.111481 21221 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:14:11.112079 21221 webserver.cc:460] Webserver started at http://127.3.6.129:44657/ using document root <none> and password file <none>
I20260925 05:14:11.112351 21221 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:11.112426 21221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:11.112599 21221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:11.113703 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "db02f9045f2e480ca6b91a1d9b99eecb"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "060400304f856d454cef9d66ba57de26"
server_key_iv: "a4ebd0d4684df4a34c561247c32d0d67"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.114166 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "db02f9045f2e480ca6b91a1d9b99eecb"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "060400304f856d454cef9d66ba57de26"
server_key_iv: "a4ebd0d4684df4a34c561247c32d0d67"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.115581 21221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:11.116520 21236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.116708 21221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:11.116812 21221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "db02f9045f2e480ca6b91a1d9b99eecb"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "060400304f856d454cef9d66ba57de26"
server_key_iv: "a4ebd0d4684df4a34c561247c32d0d67"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.116904 21221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:11.151571 21221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:11.153970 21221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:11.154171 21221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:11.154439 21221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:11.154785 21221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:11.154843 21221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.154883 21221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:11.154932 21221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.161325 21221 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:40631
I20260925 05:14:11.161401 21349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:40631 every 8 connection(s)
I20260925 05:14:11.161796 21221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:14:11.167402 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 21221
I20260925 05:14:11.167541 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:14:11.167797 3098 external_mini_cluster.cc:1428] Setting key 2c2e2a1a65af476f66c5b74c907df40c
Sep 25 05:14:11 dist-test-slave-1lvv krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:11.173482 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.162664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:46105 (local address 127.3.6.190:44579)
0925 05:14:11.162715 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.162718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.162954 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:14:11.163449 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.163450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.163461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.163526 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:11.164720 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.165317 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.166596 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.167005 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.167126 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.170231 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:11.170233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:11.170234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:11.170244 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:11.171948 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.172402 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.172404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.172404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.172474 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.172756 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.172757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.172758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.172896 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:11.172981 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.173265 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.173379 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:14:11.173753 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.162609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44579 (local address 127.3.6.129:46105)
0925 05:14:11.162804 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.162851 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.163361 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.163584 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.163587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.164225 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:11.164447 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.164453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.165427 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.165428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.166431 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.166434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.167132 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.167133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.167296 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.168234 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:11.168245 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:11.170115 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:11.172028 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:11.172036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:11.172038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:11.172283 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:11.172550 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:11.172551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:11.172551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:11.172680 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:11.172969 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:11.172976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:11.173126 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.173245 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.173441 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260925 05:14:11.174315 21350 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44579
I20260925 05:14:11.174425 21350 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:11.174649 21350 heartbeater.cc:511] Master 127.3.6.190:44579 requested a full tablet report, sending...
I20260925 05:14:11.175307 21159 ts_manager.cc:194] Registered new tserver with Master: db02f9045f2e480ca6b91a1d9b99eecb (127.3.6.129:40631)
I20260925 05:14:11.176090 21159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:46105
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:14:11.198619 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44579
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:11.285732 21355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:11.285964 21355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:11.285991 21355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:11.287226 21355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:11.287278 21355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:11.287307 21355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:14:11.288887 21355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:11.289167 21355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:11.289419 21355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:11.294704 21363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:11.294742 21360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:11.294984 21355 server_base.cc:1034] running on GCE node
W20260925 05:14:11.294713 21361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:11.295301 21355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:11.295588 21355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:11.296732 21355 hybrid_clock.cc:648] HybridClock initialized: now 1790313251296717 us; error 43 us; skew 500 ppm
Sep 25 05:14:11 dist-test-slave-1lvv krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:11.299459 21355 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:14:11.300001 21355 webserver.cc:460] Webserver started at http://127.3.6.130:44449/ using document root <none> and password file <none>
I20260925 05:14:11.300262 21355 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:11.300333 21355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:11.300482 21355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:11.301582 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "94c2c0994ca44de8be7f6077ae6e5275"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "ec74ba2bdfe316ce27e4164749ee6528"
server_key_iv: "e6ef5f522aadf22f85f2274d04fa55c2"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.301981 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "94c2c0994ca44de8be7f6077ae6e5275"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "ec74ba2bdfe316ce27e4164749ee6528"
server_key_iv: "e6ef5f522aadf22f85f2274d04fa55c2"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.303413 21355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:11.304284 21370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.304467 21355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:14:11.304596 21355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "94c2c0994ca44de8be7f6077ae6e5275"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "ec74ba2bdfe316ce27e4164749ee6528"
server_key_iv: "e6ef5f522aadf22f85f2274d04fa55c2"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.304677 21355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:11.320542 21355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:11.322961 21355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:11.323153 21355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:11.323405 21355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:11.323796 21355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:11.323866 21355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.323905 21355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:11.323932 21355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.330168 21355 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:37373
I20260925 05:14:11.330212 21483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:37373 every 8 connection(s)
I20260925 05:14:11.330567 21355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:14:11.334736 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 21355
I20260925 05:14:11.334872 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:14:11.335069 3098 external_mini_cluster.cc:1428] Setting key c65e9001f5c93ce40dce3c6d63c44f02
Sep 25 05:14:11 dist-test-slave-1lvv krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:11.341351 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.331295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44579 (local address 127.3.6.130:42643)
0925 05:14:11.331533 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.331579 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.332169 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.332451 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.332455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.333044 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:11.333245 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.333250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.334037 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.334039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.335086 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.335089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.335597 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.335598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.335736 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.336372 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:11.336380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:11.338072 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:11.339835 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:11.339843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:11.339845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:11.340136 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:11.340386 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:11.340387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:11.340387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:11.340519 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:11.340826 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:11.340830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:11.340954 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.341039 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.341172 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260925 05:14:11.341354 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.331410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:42643 (local address 127.3.6.190:44579)
0925 05:14:11.331496 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.331499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.331672 (+ 173us) server_negotiation.cc:407] Connection header received
0925 05:14:11.332286 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.332288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.332301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.332367 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:11.333356 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.333930 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.335213 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.335487 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.335584 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.338168 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:11.338171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:11.338171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:11.338180 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:11.339693 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.340231 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.340233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.340234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.340306 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.340590 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.340591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.340591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.340755 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:11.340835 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.341105 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.341212 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:14:11.342242 21484 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44579
I20260925 05:14:11.342365 21484 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:11.342600 21484 heartbeater.cc:511] Master 127.3.6.190:44579 requested a full tablet report, sending...
I20260925 05:14:11.343079 21159 ts_manager.cc:194] Registered new tserver with Master: 94c2c0994ca44de8be7f6077ae6e5275 (127.3.6.130:37373)
I20260925 05:14:11.343480 21159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:42643
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:14:11.367102 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44579
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:11.462589 21489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:11.462833 21489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:11.462860 21489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:11.464207 21489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:11.464253 21489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:11.464282 21489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:14:11.465830 21489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:44579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:11.466049 21489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:11.466238 21489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:11.471661 21494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:11.471731 21497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:11.471678 21495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:11.471920 21489 server_base.cc:1034] running on GCE node
I20260925 05:14:11.472121 21489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:11.472330 21489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:11.473510 21489 hybrid_clock.cc:648] HybridClock initialized: now 1790313251473497 us; error 29 us; skew 500 ppm
Sep 25 05:14:11 dist-test-slave-1lvv krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:11.476315 21489 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:14:11.476760 21489 webserver.cc:460] Webserver started at http://127.3.6.131:43859/ using document root <none> and password file <none>
I20260925 05:14:11.476984 21489 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:11.477028 21489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:11.477125 21489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:11.478192 21489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "afe65bcea97c4494b4b6c92613e6f12c"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "c296b99a25144ba58aa9f8c3d32ec9f8"
server_key_iv: "ab4e8c02759d4f3218f9e0009249ba30"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.478513 21489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "afe65bcea97c4494b4b6c92613e6f12c"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "c296b99a25144ba58aa9f8c3d32ec9f8"
server_key_iv: "ab4e8c02759d4f3218f9e0009249ba30"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.479925 21489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:11.480640 21504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.480801 21489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:11.480898 21489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "afe65bcea97c4494b4b6c92613e6f12c"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "c296b99a25144ba58aa9f8c3d32ec9f8"
server_key_iv: "ab4e8c02759d4f3218f9e0009249ba30"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.480958 21489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:11.501153 21489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:11.503633 21489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:11.503861 21489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:11.504096 21489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:11.504451 21489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:11.504487 21489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.504511 21489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:11.504565 21489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.511031 21489 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:36293
I20260925 05:14:11.511116 21617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:36293 every 8 connection(s)
I20260925 05:14:11.511511 21489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:14:11.514657 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 21489
I20260925 05:14:11.514791 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:14:11.515008 3098 external_mini_cluster.cc:1428] Setting key e8bc93b00f3e618fa083d2e9f904e3d2
Sep 25 05:14:11 dist-test-slave-1lvv krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:11.522202 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.512464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:49035 (local address 127.3.6.190:44579)
0925 05:14:11.512528 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.512531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.512806 (+ 275us) server_negotiation.cc:407] Connection header received
0925 05:14:11.513359 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.513361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.513374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.513436 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:11.514456 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.515088 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.516322 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.516539 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.516631 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.518947 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:11.518949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:11.518949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:11.518956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:11.520650 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.521203 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.521204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.521205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.521286 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.521593 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.521595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.521595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.521735 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:11.521794 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.522062 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.522132 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:14:11.522425 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.512451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:44579 (local address 127.3.6.131:49035)
0925 05:14:11.512674 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.512716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.513246 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.513529 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.513532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.514135 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:11.514346 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.514351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.515178 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.515180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.516216 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.516219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.516642 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.516643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.516772 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.517266 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:11.517275 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:11.518846 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:11.520756 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:11.520765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:11.520768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:11.521110 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:11.521387 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:11.521388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:11.521389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:11.521530 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:11.521801 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:11.521806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:11.521941 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.522031 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.522165 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":126,"threads_started":1}
I20260925 05:14:11.522941 21618 heartbeater.cc:347] Connected to a master server at 127.3.6.190:44579
I20260925 05:14:11.523075 21618 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:11.523303 21618 heartbeater.cc:511] Master 127.3.6.190:44579 requested a full tablet report, sending...
I20260925 05:14:11.523813 21159 ts_manager.cc:194] Registered new tserver with Master: afe65bcea97c4494b4b6c92613e6f12c (127.3.6.131:36293)
I20260925 05:14:11.524370 21159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:49035
I20260925 05:14:11.526942 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:14:11.534912 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.528426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.3.6.190:44579)
0925 05:14:11.528490 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.528493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.528680 (+ 187us) server_negotiation.cc:407] Connection header received
0925 05:14:11.528719 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.528720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.528730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.528776 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:11.529321 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.529809 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.530505 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.530688 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.530739 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.532170 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:11.532171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:11.532171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:11.532179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:11.533637 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.534070 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.534071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.534071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.534126 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:11.534363 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:11.534364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:11.534364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:11.534509 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:11.534597 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.534735 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.534817 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:14:11.536715 21159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:14:11.538319 21159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:14:11.545696 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.541614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.3.6.129:40631)
0925 05:14:11.541978 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.541983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.541993 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:11.542045 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.542048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.542063 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.542150 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:11.542274 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.543341 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.544752 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.545304 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.545390 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.545454 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.545572 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.545616 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":94,"threads_started":1}
I20260925 05:14:11.545686 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.540936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:36293 (local address 127.0.0.1:37862)
0925 05:14:11.541061 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.541104 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.541169 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.541736 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.541740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.541744 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:11.541860 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.541863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.543035 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.543036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.544530 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.544532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.545324 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.545325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.545413 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.545427 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.545498 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.545568 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":33,"thread_start_us":76,"threads_started":1}
I20260925 05:14:11.545884 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.541331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.3.6.131:36293)
0925 05:14:11.541517 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.541523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.541535 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:11.541578 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.541581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.541599 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.541950 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:11.542023 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.542892 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.544627 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.545228 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.545293 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.545336 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.545731 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.545786 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260925 05:14:11.545686 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.541176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40631 (local address 127.0.0.1:48940)
0925 05:14:11.541807 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.541817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.541859 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.542167 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.542168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.542170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:11.542210 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.542211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.543448 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.543449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.544663 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.544666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.545409 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.545410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.545473 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.545477 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.545539 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.545583 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":76,"threads_started":1}
I20260925 05:14:11.546649 21552 tablet_service.cc:1467] Processing CreateTablet for tablet b632decfdb7440f9a508660bf60090e5 (DEFAULT_TABLE table=test-table [id=de6e8101cfd84f7281500a73353d19aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:11.546648 21284 tablet_service.cc:1467] Processing CreateTablet for tablet b632decfdb7440f9a508660bf60090e5 (DEFAULT_TABLE table=test-table [id=de6e8101cfd84f7281500a73353d19aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:11.547070 21552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b632decfdb7440f9a508660bf60090e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:11.547070 21284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b632decfdb7440f9a508660bf60090e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:11.547219 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.541027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:37373 (local address 127.0.0.1:42292)
0925 05:14:11.541185 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.541197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.541257 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.541826 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.541827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.541829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:11.541896 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.541898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.543297 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.543297 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.544530 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.544532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.546962 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.546963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.547042 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.547057 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.547084 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.547142 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":26,"thread_start_us":116,"threads_started":1}
I20260925 05:14:11.547569 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.541376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.3.6.130:37373)
0925 05:14:11.541625 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.541631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.541643 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:11.541688 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.541691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.541707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.541979 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:11.542059 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.543188 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.545402 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.546825 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.547343 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.547421 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.547450 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.547498 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":113,"threads_started":1}
I20260925 05:14:11.548470 21418 tablet_service.cc:1467] Processing CreateTablet for tablet b632decfdb7440f9a508660bf60090e5 (DEFAULT_TABLE table=test-table [id=de6e8101cfd84f7281500a73353d19aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:11.548832 21418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b632decfdb7440f9a508660bf60090e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:11.550678 21635 tablet_bootstrap.cc:492] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Bootstrap starting.
I20260925 05:14:11.550980 21636 tablet_bootstrap.cc:492] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb: Bootstrap starting.
I20260925 05:14:11.551396 21635 tablet_bootstrap.cc:654] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:11.551613 21636 tablet_bootstrap.cc:654] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:11.552080 21635 log.cc:824] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:11.552076 21637 tablet_bootstrap.cc:492] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Bootstrap starting.
I20260925 05:14:11.552279 21636 log.cc:824] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:11.552920 21637 tablet_bootstrap.cc:654] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:11.553030 21635 tablet_bootstrap.cc:492] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: No bootstrap required, opened a new log
I20260925 05:14:11.553109 21635 ts_tablet_manager.cc:1397] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:14:11.553211 21636 tablet_bootstrap.cc:492] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb: No bootstrap required, opened a new log
I20260925 05:14:11.553303 21636 ts_tablet_manager.cc:1397] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:14:11.553635 21637 log.cc:824] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:11.554438 21637 tablet_bootstrap.cc:492] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: No bootstrap required, opened a new log
I20260925 05:14:11.554509 21637 ts_tablet_manager.cc:1397] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:14:11.554893 21635 raft_consensus.cc:348] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.555073 21635 raft_consensus.cc:374] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:11.555119 21635 raft_consensus.cc:729] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe65bcea97c4494b4b6c92613e6f12c, State: Initialized, Role: FOLLOWER
I20260925 05:14:11.555240 21635 consensus_queue.cc:260] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.555428 21636 raft_consensus.cc:348] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.555550 21635 ts_tablet_manager.cc:1428] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:14:11.555608 21636 raft_consensus.cc:374] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:11.555552 21618 heartbeater.cc:503] Master 127.3.6.190:44579 was elected leader, sending a full tablet report...
I20260925 05:14:11.555651 21636 raft_consensus.cc:729] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db02f9045f2e480ca6b91a1d9b99eecb, State: Initialized, Role: FOLLOWER
I20260925 05:14:11.555794 21636 consensus_queue.cc:260] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.556085 21636 ts_tablet_manager.cc:1428] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:14:11.556231 21350 heartbeater.cc:503] Master 127.3.6.190:44579 was elected leader, sending a full tablet report...
I20260925 05:14:11.556628 21637 raft_consensus.cc:348] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.556824 21637 raft_consensus.cc:374] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:11.556879 21637 raft_consensus.cc:729] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c2c0994ca44de8be7f6077ae6e5275, State: Initialized, Role: FOLLOWER
I20260925 05:14:11.557003 21637 consensus_queue.cc:260] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.557328 21637 ts_tablet_manager.cc:1428] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:14:11.557389 21484 heartbeater.cc:503] Master 127.3.6.190:44579 was elected leader, sending a full tablet report...
W20260925 05:14:11.581171 21485 tablet.cc:2369] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:14:11.662585 21351 tablet.cc:2369] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:14:11.757194 21643 raft_consensus.cc:482] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:11.757333 21643 raft_consensus.cc:504] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.757767 21643 leader_election.cc:288] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afe65bcea97c4494b4b6c92613e6f12c (127.3.6.131:36293), db02f9045f2e480ca6b91a1d9b99eecb (127.3.6.129:40631)
I20260925 05:14:11.760809 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.758020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:36293 (local address 127.3.6.130:40059)
0925 05:14:11.758091 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.758107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.758184 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.758365 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.758366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.758368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:11.758441 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.758443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.759108 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.759109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.759921 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.759923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.760524 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.760525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.760611 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.760614 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.760684 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.760731 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 05:14:11.760809 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.758112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:40059 (local address 127.3.6.131:36293)
0925 05:14:11.758192 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.758195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.758229 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:14:11.758258 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.758259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.758276 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.758326 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:11.758528 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.759007 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.760028 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.760428 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.760512 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.760524 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.760689 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.760729 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:14:11.761112 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.758246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:42407 (local address 127.3.6.129:40631)
0925 05:14:11.758320 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.758322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.758476 (+ 154us) server_negotiation.cc:407] Connection header received
0925 05:14:11.758547 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.758548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.758561 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.758640 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:11.758803 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.759273 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.760312 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.760688 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.760773 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.760784 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.760984 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.761046 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:14:11.761247 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632decfdb7440f9a508660bf60090e5" candidate_uuid: "94c2c0994ca44de8be7f6077ae6e5275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe65bcea97c4494b4b6c92613e6f12c" is_pre_election: true
I20260925 05:14:11.761379 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.758178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40631 (local address 127.3.6.130:42407)
0925 05:14:11.758374 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.758386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:11.758483 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:11.758644 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:11.758645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:11.758646 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:11.758727 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.758728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.759389 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.759390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.760209 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:11.760212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.760788 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.760789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:11.760864 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.760869 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:14:11.761248 (+ 379us) client_negotiation.cc:240] Negotiation successful
0925 05:14:11.761295 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":126,"threads_started":1}
I20260925 05:14:11.761451 21572 raft_consensus.cc:2454] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94c2c0994ca44de8be7f6077ae6e5275 in term 0.
I20260925 05:14:11.761744 21372 leader_election.cc:302] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94c2c0994ca44de8be7f6077ae6e5275, afe65bcea97c4494b4b6c92613e6f12c; no voters:
I20260925 05:14:11.761754 21304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632decfdb7440f9a508660bf60090e5" candidate_uuid: "94c2c0994ca44de8be7f6077ae6e5275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" is_pre_election: true
I20260925 05:14:11.761929 21304 raft_consensus.cc:2454] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94c2c0994ca44de8be7f6077ae6e5275 in term 0.
I20260925 05:14:11.761919 21643 raft_consensus.cc:2781] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:14:11.761994 21643 raft_consensus.cc:482] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:14:11.762022 21643 raft_consensus.cc:3037] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 0 FOLLOWER]: Advancing to term 1
W20260925 05:14:11.762167 21619 tablet.cc:2369] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:14:11.762867 21643 raft_consensus.cc:504] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.763020 21643 leader_election.cc:288] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [CANDIDATE]: Term 1 election: Requested vote from peers afe65bcea97c4494b4b6c92613e6f12c (127.3.6.131:36293), db02f9045f2e480ca6b91a1d9b99eecb (127.3.6.129:40631)
I20260925 05:14:11.763195 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632decfdb7440f9a508660bf60090e5" candidate_uuid: "94c2c0994ca44de8be7f6077ae6e5275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe65bcea97c4494b4b6c92613e6f12c"
I20260925 05:14:11.763275 21572 raft_consensus.cc:3037] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:11.763235 21304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b632decfdb7440f9a508660bf60090e5" candidate_uuid: "94c2c0994ca44de8be7f6077ae6e5275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db02f9045f2e480ca6b91a1d9b99eecb"
I20260925 05:14:11.763340 21304 raft_consensus.cc:3037] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:11.764168 21572 raft_consensus.cc:2454] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c2c0994ca44de8be7f6077ae6e5275 in term 1.
I20260925 05:14:11.764336 21304 raft_consensus.cc:2454] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c2c0994ca44de8be7f6077ae6e5275 in term 1.
I20260925 05:14:11.764365 21372 leader_election.cc:302] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94c2c0994ca44de8be7f6077ae6e5275, afe65bcea97c4494b4b6c92613e6f12c; no voters:
I20260925 05:14:11.764488 21643 raft_consensus.cc:2781] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:11.764708 21643 raft_consensus.cc:686] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 1 LEADER]: Becoming Leader. State: Replica: 94c2c0994ca44de8be7f6077ae6e5275, State: Running, Role: LEADER
I20260925 05:14:11.764839 21643 consensus_queue.cc:237] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } }
I20260925 05:14:11.766047 21158 catalog_manager.cc:5495] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94c2c0994ca44de8be7f6077ae6e5275 (127.3.6.130). New cstate: current_term: 1 leader_uuid: "94c2c0994ca44de8be7f6077ae6e5275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c2c0994ca44de8be7f6077ae6e5275" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37373 } health_report { overall_health: HEALTHY } } }
I20260925 05:14:11.775012 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.772715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.3.6.130:37373)
0925 05:14:11.772783 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.772785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.772796 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:14:11.772840 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.772841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.772860 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.772926 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:11.773059 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.773536 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.774263 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.774473 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.774525 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.774678 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:11.774768 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:11.774839 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.774881 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.774889 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:14:11.777158 21304 raft_consensus.cc:1261] T b632decfdb7440f9a508660bf60090e5 P db02f9045f2e480ca6b91a1d9b99eecb [term 1 FOLLOWER]: Refusing update from remote peer 94c2c0994ca44de8be7f6077ae6e5275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:11.777159 21572 raft_consensus.cc:1261] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 1 FOLLOWER]: Refusing update from remote peer 94c2c0994ca44de8be7f6077ae6e5275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:11.777658 21645 consensus_queue.cc:1035] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db02f9045f2e480ca6b91a1d9b99eecb" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 40631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:11.777810 21643 consensus_queue.cc:1035] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe65bcea97c4494b4b6c92613e6f12c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:11.780340 21648 mvcc.cc:205] Tried to move back new op lower bound from 7333123079275995136 to 7333123079229394944. Current Snapshot: MvccSnapshot[applied={T|T < 7333123079275995136}]
I20260925 05:14:11.782868 21159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:11.782963 21159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:11.783808 21159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a113e2ed9c654c439bf915d47166883e: Sending DeleteTablet for 3 replicas of tablet b632decfdb7440f9a508660bf60090e5
I20260925 05:14:11.784166 21552 tablet_service.cc:1514] Processing DeleteTablet for tablet b632decfdb7440f9a508660bf60090e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:11 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:37862
I20260925 05:14:11.784184 21418 tablet_service.cc:1514] Processing DeleteTablet for tablet b632decfdb7440f9a508660bf60090e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:11 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:42292
I20260925 05:14:11.784469 21659 tablet_replica.cc:330] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: stopping tablet replica
I20260925 05:14:11.784467 21284 tablet_service.cc:1514] Processing DeleteTablet for tablet b632decfdb7440f9a508660bf60090e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:11 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:48940
I20260925 05:14:11.784647 21659 raft_consensus.cc:2229] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:14:11.784739 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 21221
I20260925 05:14:11.786787 21659 raft_consensus.cc:2258] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:11.787096 21658 tablet_replica.cc:330] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: stopping tablet replica
I20260925 05:14:11.787223 21658 raft_consensus.cc:2229] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:11.787346 21658 raft_consensus.cc:2258] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:11.787364 21659 ts_tablet_manager.cc:1905] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:11.787905 21658 ts_tablet_manager.cc:1905] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:11.789242 21659 ts_tablet_manager.cc:1918] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:11.789346 21659 log.cc:1196] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/b632decfdb7440f9a508660bf60090e5
I20260925 05:14:11.789624 21659 ts_tablet_manager.cc:1939] T b632decfdb7440f9a508660bf60090e5 P 94c2c0994ca44de8be7f6077ae6e5275: Deleting consensus metadata
I20260925 05:14:11.789966 21658 ts_tablet_manager.cc:1918] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:11.790055 21658 log.cc:1196] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/b632decfdb7440f9a508660bf60090e5
I20260925 05:14:11.790141 21143 catalog_manager.cc:4841] TS 94c2c0994ca44de8be7f6077ae6e5275 (127.3.6.130:37373): tablet b632decfdb7440f9a508660bf60090e5 (table test-table [id=de6e8101cfd84f7281500a73353d19aa]) successfully deleted
I20260925 05:14:11.790352 21658 ts_tablet_manager.cc:1939] T b632decfdb7440f9a508660bf60090e5 P afe65bcea97c4494b4b6c92613e6f12c: Deleting consensus metadata
I20260925 05:14:11.790802 21144 catalog_manager.cc:4841] TS afe65bcea97c4494b4b6c92613e6f12c (127.3.6.131:36293): tablet b632decfdb7440f9a508660bf60090e5 (table test-table [id=de6e8101cfd84f7281500a73353d19aa]) successfully deleted
W20260925 05:14:11.792263 21144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:40631 (error 108)
I20260925 05:14:11.792469 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 21355
I20260925 05:14:11.793524 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.793318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:40631 (local address 127.0.0.1:48948)
0925 05:14:11.793406 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:11.793487 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:14:11.793687 21144 catalog_manager.cc:4584] TS db02f9045f2e480ca6b91a1d9b99eecb (127.3.6.129:40631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b632decfdb7440f9a508660bf60090e5: Network error: Client connection negotiation failed: client connection to 127.3.6.129:40631: connect: Connection refused (error 111)
I20260925 05:14:11.798475 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 21489
I20260925 05:14:11.803635 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 21127
2026-09-25T05:14:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2186 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T05:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:11Z Disabled control of system clock
I20260925 05:14:11.834050 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:37043
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:36875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:14:11.918243 21665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:11.918571 21665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:11.918597 21665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:11.919834 21665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:11.919880 21665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:11.919893 21665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:11.919906 21665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:11.919919 21665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:11.921288 21665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:37043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:11.921509 21665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:11.921768 21665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:11.926853 21670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:11.926939 21673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:11.926908 21671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:11.927132 21665 server_base.cc:1034] running on GCE node
I20260925 05:14:11.927412 21665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:11.927826 21665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:11.928990 21665 hybrid_clock.cc:648] HybridClock initialized: now 1790313251928970 us; error 43 us; skew 500 ppm
I20260925 05:14:11.938068 21665 webserver.cc:460] Webserver started at http://127.3.6.190:36967/ using document root <none> and password file <none>
I20260925 05:14:11.938400 21665 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:11.938455 21665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:11.938627 21665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:11.939910 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "d0199350292d4e66bab52d1ba1cc0e11"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "ae33102408d154e7b45bfc5686a37d66"
server_key_iv: "3aa30a04d01eb8c8ac08a88b15aadf0a"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.940312 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "d0199350292d4e66bab52d1ba1cc0e11"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "ae33102408d154e7b45bfc5686a37d66"
server_key_iv: "3aa30a04d01eb8c8ac08a88b15aadf0a"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.941843 21665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:14:11.942724 21679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:11.942950 21665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:11.943079 21665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "d0199350292d4e66bab52d1ba1cc0e11"
format_stamp: "Formatted at 2026-09-25 05:14:11 on dist-test-slave-1lvv"
server_key: "ae33102408d154e7b45bfc5686a37d66"
server_key_iv: "3aa30a04d01eb8c8ac08a88b15aadf0a"
server_key_version: "encryptionkey@0"
I20260925 05:14:11.943168 21665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:11.955245 21665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:11.955653 21665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:11.955844 21665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:11.961621 21665 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:37043
I20260925 05:14:11.961658 21731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:37043 every 8 connection(s)
I20260925 05:14:11.962062 21665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:11.963110 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:11.965620 21732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11: Bootstrap starting.
I20260925 05:14:11.966337 21732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:11.966753 21732 log.cc:824] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:11.968304 21732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11: No bootstrap required, opened a new log
I20260925 05:14:11.969288 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 21665
I20260925 05:14:11.969383 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:11.969630 3098 external_mini_cluster.cc:1428] Setting key 84193a0e22fb7ecd9e71d67cac89574c
I20260925 05:14:11.970600 21732 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0199350292d4e66bab52d1ba1cc0e11" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37043 } }
I20260925 05:14:11.970789 21732 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:11.970829 21732 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0199350292d4e66bab52d1ba1cc0e11, State: Initialized, Role: FOLLOWER
I20260925 05:14:11.970988 21732 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0199350292d4e66bab52d1ba1cc0e11" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37043 } }
I20260925 05:14:11.971086 21732 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:11.971138 21732 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:11.971191 21732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:11.971872 21732 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0199350292d4e66bab52d1ba1cc0e11" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37043 } }
I20260925 05:14:11.972002 21732 leader_election.cc:302] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0199350292d4e66bab52d1ba1cc0e11; no voters:
I20260925 05:14:11.972189 21732 leader_election.cc:288] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:11.972236 21737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:11.972364 21737 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [term 1 LEADER]: Becoming Leader. State: Replica: d0199350292d4e66bab52d1ba1cc0e11, State: Running, Role: LEADER
I20260925 05:14:11.972481 21732 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:11.972458 21737 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0199350292d4e66bab52d1ba1cc0e11" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37043 } }
I20260925 05:14:11.972836 21737 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0199350292d4e66bab52d1ba1cc0e11. Latest consensus state: current_term: 1 leader_uuid: "d0199350292d4e66bab52d1ba1cc0e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0199350292d4e66bab52d1ba1cc0e11" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37043 } } }
I20260925 05:14:11.972851 21739 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0199350292d4e66bab52d1ba1cc0e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0199350292d4e66bab52d1ba1cc0e11" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37043 } } }
I20260925 05:14:11.972926 21737 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:11.972950 21739 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:11.973321 21743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:11.974494 21743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:11.976713 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:11.970246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.3.6.190:37043)
0925 05:14:11.970515 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:11.970526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:11.970563 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:14:11.970684 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:11.970691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:11.970765 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:11.970922 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:11.971230 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.973158 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.974338 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:11.974874 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:11.974967 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:11.975854 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:11.975858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:11.975859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:11.976080 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:11.976126 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:11.976142 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:11.976212 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:11.976451 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 05:14:11.976538 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260925 05:14:11.977192 21743 catalog_manager.cc:1347] Generated new cluster ID: f61cedb03d2844ee97d236cd2ea9e3b4
I20260925 05:14:11.977246 21743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:11.982933 21743 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:11.983693 21743 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:11.989669 21743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11: Generated new TSK 0
I20260925 05:14:11.989894 21743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:11.993636 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37043
--builtin_ntp_servers=127.3.6.148:36875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:14:12.088521 21756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:12.088796 21756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:12.088838 21756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:12.090054 21756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:12.090130 21756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:12.090176 21756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:14:12.091566 21756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:12.091883 21756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:12.092219 21756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:14:12.098328 21756 server_base.cc:1034] running on GCE node
W20260925 05:14:12.098295 21761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:12.098402 21764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:12.098301 21762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:12.098781 21756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:12.099049 21756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:12.100226 21756 hybrid_clock.cc:648] HybridClock initialized: now 1790313252100211 us; error 33 us; skew 500 ppm
I20260925 05:14:12.101645 21756 webserver.cc:460] Webserver started at http://127.3.6.129:38707/ using document root <none> and password file <none>
I20260925 05:14:12.101897 21756 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:12.101967 21756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:12.102121 21756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:12.103230 21756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "c4925ab9674540bca2181fc6a3080569"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "f6c4bd2e65845a59d323bd318d17b8ec"
server_key_iv: "5293839366b6917c49b9c6c9b0184a3c"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.103616 21756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "c4925ab9674540bca2181fc6a3080569"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "f6c4bd2e65845a59d323bd318d17b8ec"
server_key_iv: "5293839366b6917c49b9c6c9b0184a3c"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.105073 21756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:12.105861 21770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.106025 21756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:12.106125 21756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "c4925ab9674540bca2181fc6a3080569"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "f6c4bd2e65845a59d323bd318d17b8ec"
server_key_iv: "5293839366b6917c49b9c6c9b0184a3c"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.106202 21756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:12.133670 21756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:12.134045 21756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:12.134225 21756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:12.134507 21756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:12.134877 21756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:12.134912 21756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.134935 21756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:12.134984 21756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.141187 21756 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:33245
I20260925 05:14:12.141243 21883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:33245 every 8 connection(s)
I20260925 05:14:12.141618 21756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:14:12.143815 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 21756
I20260925 05:14:12.143957 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:14:12.144156 3098 external_mini_cluster.cc:1428] Setting key dcee97044fae7073f909971ba73d92c6
I20260925 05:14:12.145241 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37043
--builtin_ntp_servers=127.3.6.148:36875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:14:12.149858 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.142383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:44575 (local address 127.3.6.190:37043)
0925 05:14:12.142438 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.142440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.142704 (+ 264us) server_negotiation.cc:407] Connection header received
0925 05:14:12.143295 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.143296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.143312 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.143361 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:12.144242 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.144892 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.148046 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.148401 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.148530 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.149479 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:12.149481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:12.149482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:12.149488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:12.149511 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:12.149517 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:12.149589 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.149680 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.149724 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:14:12.149968 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.142358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37043 (local address 127.3.6.129:44575)
0925 05:14:12.142566 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.142604 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.143162 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.143450 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.143454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.143467 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:12.144138 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.144145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.146894 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.146895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.147941 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.147944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.148550 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.148551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.148687 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.149314 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:12.149315 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:12.149349 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:12.149378 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:12.149595 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:12.149600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:12.149603 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.149660 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.149754 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260925 05:14:12.150578 21884 heartbeater.cc:347] Connected to a master server at 127.3.6.190:37043
I20260925 05:14:12.150681 21884 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:12.150887 21884 heartbeater.cc:511] Master 127.3.6.190:37043 requested a full tablet report, sending...
I20260925 05:14:12.151521 21696 ts_manager.cc:194] Registered new tserver with Master: c4925ab9674540bca2181fc6a3080569 (127.3.6.129:33245)
I20260925 05:14:12.152599 21696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.129:44575
W20260925 05:14:12.230324 21887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:12.230597 21887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:12.230644 21887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:12.231803 21887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:12.231837 21887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:12.231863 21887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:14:12.233148 21887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:12.233336 21887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:12.233597 21887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:12.238816 21893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:12.238835 21895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:12.238859 21892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:12.238912 21887 server_base.cc:1034] running on GCE node
I20260925 05:14:12.239286 21887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:12.239585 21887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:12.240753 21887 hybrid_clock.cc:648] HybridClock initialized: now 1790313252240726 us; error 60 us; skew 500 ppm
I20260925 05:14:12.242053 21887 webserver.cc:460] Webserver started at http://127.3.6.130:35677/ using document root <none> and password file <none>
I20260925 05:14:12.242370 21887 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:12.242452 21887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:12.242610 21887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:12.243723 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "74f808adfb9d4708bd915703d5e12870"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "639a5325f3b93a3e55a5990b1a77bcd4"
server_key_iv: "f1c786c0caa9f767147706d587e615f1"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.244170 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "74f808adfb9d4708bd915703d5e12870"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "639a5325f3b93a3e55a5990b1a77bcd4"
server_key_iv: "f1c786c0caa9f767147706d587e615f1"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.245747 21887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:14:12.246577 21901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.246811 21887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:12.246912 21887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "74f808adfb9d4708bd915703d5e12870"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "639a5325f3b93a3e55a5990b1a77bcd4"
server_key_iv: "f1c786c0caa9f767147706d587e615f1"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.246999 21887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:12.260833 21887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:12.261198 21887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:12.261371 21887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:12.261636 21887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:12.262001 21887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:12.262058 21887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.262094 21887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:12.262138 21887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.268015 21887 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:39257
I20260925 05:14:12.268096 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:39257 every 8 connection(s)
I20260925 05:14:12.268467 21887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:14:12.271324 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 21887
I20260925 05:14:12.271471 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:14:12.271742 3098 external_mini_cluster.cc:1428] Setting key 49b0790fd99310147f8fb321305d96fe
I20260925 05:14:12.272962 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37043
--builtin_ntp_servers=127.3.6.148:36875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:14:12.275439 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.269307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:35765 (local address 127.3.6.190:37043)
0925 05:14:12.269432 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.269436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.269670 (+ 234us) server_negotiation.cc:407] Connection header received
0925 05:14:12.270455 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.270456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.270469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.270525 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:12.271428 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.272196 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.273787 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.274055 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.274110 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.275130 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:12.275132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:12.275132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:12.275138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:12.275160 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:12.275166 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:12.275229 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.275298 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.275331 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:14:12.275589 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.269285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37043 (local address 127.3.6.130:35765)
0925 05:14:12.269536 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.269571 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.270323 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.270602 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.270607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.270618 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:12.271325 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.271332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.272311 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.272312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.273656 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.273659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.274137 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.274139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.274306 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.274980 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:12.274980 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:12.275011 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:12.275038 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:12.275235 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:12.275240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:12.275242 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.275297 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.275385 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260925 05:14:12.276048 22015 heartbeater.cc:347] Connected to a master server at 127.3.6.190:37043
I20260925 05:14:12.276163 22015 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:12.276374 22015 heartbeater.cc:511] Master 127.3.6.190:37043 requested a full tablet report, sending...
I20260925 05:14:12.276787 21696 ts_manager.cc:194] Registered new tserver with Master: 74f808adfb9d4708bd915703d5e12870 (127.3.6.130:39257)
I20260925 05:14:12.277185 21696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.130:35765
W20260925 05:14:12.356973 22018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:12.357193 22018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:12.357215 22018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:12.358381 22018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:12.358431 22018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:12.358458 22018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:14:12.359915 22018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:36875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:37043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:12.360177 22018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:12.360507 22018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:12.365826 22024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:12.365789 22026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:12.365813 22023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:12.366266 22018 server_base.cc:1034] running on GCE node
I20260925 05:14:12.366508 22018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:12.366781 22018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:12.367945 22018 hybrid_clock.cc:648] HybridClock initialized: now 1790313252367927 us; error 33 us; skew 500 ppm
I20260925 05:14:12.369336 22018 webserver.cc:460] Webserver started at http://127.3.6.131:39785/ using document root <none> and password file <none>
I20260925 05:14:12.369606 22018 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:12.369679 22018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:12.369833 22018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:12.370939 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "a30bb68627694ea59086e7756bf5e7bb"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "731791e9dcf909fb1febd1307b17037d"
server_key_iv: "720b71f36d223edf1b58d07b73b60067"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.371316 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "a30bb68627694ea59086e7756bf5e7bb"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "731791e9dcf909fb1febd1307b17037d"
server_key_iv: "720b71f36d223edf1b58d07b73b60067"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.372840 22018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:12.373659 22032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.373835 22018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:14:12.373936 22018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "a30bb68627694ea59086e7756bf5e7bb"
format_stamp: "Formatted at 2026-09-25 05:14:12 on dist-test-slave-1lvv"
server_key: "731791e9dcf909fb1febd1307b17037d"
server_key_iv: "720b71f36d223edf1b58d07b73b60067"
server_key_version: "encryptionkey@0"
I20260925 05:14:12.374022 22018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:12.386118 22018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:12.386493 22018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:12.386662 22018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:12.386930 22018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:12.387312 22018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:12.387370 22018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.387416 22018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:12.387446 22018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:12.394125 22018 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:45901
I20260925 05:14:12.394172 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:45901 every 8 connection(s)
I20260925 05:14:12.394556 22018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:14:12.399837 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22018
I20260925 05:14:12.399976 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:14:12.400169 3098 external_mini_cluster.cc:1428] Setting key 593dbbc3f6d323d135c1fb1a513d2957
I20260925 05:14:12.400228 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.395414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:56623 (local address 127.3.6.190:37043)
0925 05:14:12.395503 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.395505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.395804 (+ 299us) server_negotiation.cc:407] Connection header received
0925 05:14:12.396352 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.396353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.396364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.396429 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:12.397137 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.397637 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.398663 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.398853 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.398911 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.399894 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:12.399895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:12.399895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:12.399901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:12.399917 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:12.399921 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:12.399997 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.400107 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.400144 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:14:12.400513 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.395423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:37043 (local address 127.3.6.131:56623)
0925 05:14:12.395621 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.395666 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.396252 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.396503 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.396507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.396516 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:12.397048 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.397053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.397729 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.397730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.398572 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.398574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.398945 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.398946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.399069 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.399685 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:12.399687 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:12.399729 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:12.399786 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:12.400016 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:12.400025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:12.400029 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.400097 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.400216 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260925 05:14:12.400966 22146 heartbeater.cc:347] Connected to a master server at 127.3.6.190:37043
I20260925 05:14:12.401101 22146 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:12.401327 22146 heartbeater.cc:511] Master 127.3.6.190:37043 requested a full tablet report, sending...
I20260925 05:14:12.401696 21696 ts_manager.cc:194] Registered new tserver with Master: a30bb68627694ea59086e7756bf5e7bb (127.3.6.131:45901)
I20260925 05:14:12.402100 21696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.131:56623
I20260925 05:14:12.412452 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:14:12.417059 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.413986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.3.6.190:37043)
0925 05:14:12.414046 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.414048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.414148 (+ 100us) server_negotiation.cc:407] Connection header received
0925 05:14:12.414190 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.414191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.414203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.414258 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:12.414430 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.414928 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.415689 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.415985 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.416053 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.416801 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:12.416802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:12.416802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:12.416806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:12.416819 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:12.416823 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:12.416898 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.416982 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.416994 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:14:12.418751 21696 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:14:12.420444 21696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:14:12.426936 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.422666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:33245 (local address 127.0.0.1:45240)
0925 05:14:12.423277 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.423285 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.423327 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.423596 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.423596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.423598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:12.423635 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.423636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.424591 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.424591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.425871 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.425873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.426602 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.426603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.426685 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.426700 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.426780 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.426846 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":5,"thread_start_us":117,"threads_started":1}
I20260925 05:14:12.426930 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.422921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.3.6.129:33245)
0925 05:14:12.423098 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.423101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.423390 (+ 289us) server_negotiation.cc:407] Connection header received
0925 05:14:12.423436 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.423439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.423455 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.423528 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:12.423714 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.424468 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.425959 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.426496 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.426583 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.426685 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.426795 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.426838 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260925 05:14:12.427667 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.422649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:39257 (local address 127.0.0.1:46652)
0925 05:14:12.423140 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.423148 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.423193 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.423445 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.423446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.423449 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:12.423504 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.423506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.425639 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.425640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.426430 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.426433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.427392 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.427393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.427461 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.427466 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.427533 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.427576 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260925 05:14:12.428061 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.422649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:45901 (local address 127.0.0.1:46264)
0925 05:14:12.422853 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.422895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.422968 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.423287 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.423291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.423294 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:12.423373 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.423375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.424781 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.424782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.426766 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.426767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.427787 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.427788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.427859 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.427862 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.427948 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.427992 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260925 05:14:12.428169 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.422821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.3.6.131:45901)
0925 05:14:12.423067 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.423073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.423085 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:12.423129 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.423131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.423146 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.423267 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:12.423479 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.424697 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.426855 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.427579 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.427640 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.427724 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.428032 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.428073 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260925 05:14:12.428278 21818 tablet_service.cc:1467] Processing CreateTablet for tablet 77d4bc6042674e72b8f7f628d228abf0 (DEFAULT_TABLE table=test-table [id=65c6dc0b64ea459b8248b9679c9b7c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:12.428764 21818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d4bc6042674e72b8f7f628d228abf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:12.428817 22080 tablet_service.cc:1467] Processing CreateTablet for tablet 77d4bc6042674e72b8f7f628d228abf0 (DEFAULT_TABLE table=test-table [id=65c6dc0b64ea459b8248b9679c9b7c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:12.429188 22080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d4bc6042674e72b8f7f628d228abf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:12.431710 22160 tablet_bootstrap.cc:492] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569: Bootstrap starting.
I20260925 05:14:12.431917 22161 tablet_bootstrap.cc:492] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Bootstrap starting.
I20260925 05:14:12.432057 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.422861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.3.6.130:39257)
0925 05:14:12.423081 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.423085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.423219 (+ 134us) server_negotiation.cc:407] Connection header received
0925 05:14:12.423266 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.423268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.423283 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.423359 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:12.423600 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.424697 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.426521 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.427295 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.427371 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.427465 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.427549 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.431959 (+ 4410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
I20260925 05:14:12.432495 22160 tablet_bootstrap.cc:654] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:12.432627 22161 tablet_bootstrap.cc:654] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:12.432839 22160 log.cc:824] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:12.432770 21949 tablet_service.cc:1467] Processing CreateTablet for tablet 77d4bc6042674e72b8f7f628d228abf0 (DEFAULT_TABLE table=test-table [id=65c6dc0b64ea459b8248b9679c9b7c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:12.432950 22161 log.cc:824] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:12.433137 21949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d4bc6042674e72b8f7f628d228abf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:12.433677 22160 tablet_bootstrap.cc:492] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569: No bootstrap required, opened a new log
I20260925 05:14:12.433756 22160 ts_tablet_manager.cc:1397] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:14:12.436116 22161 tablet_bootstrap.cc:492] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: No bootstrap required, opened a new log
I20260925 05:14:12.436213 22161 ts_tablet_manager.cc:1397] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 05:14:12.436436 22160 raft_consensus.cc:348] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.436615 22160 raft_consensus.cc:374] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:12.436653 22160 raft_consensus.cc:729] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4925ab9674540bca2181fc6a3080569, State: Initialized, Role: FOLLOWER
I20260925 05:14:12.436692 22164 tablet_bootstrap.cc:492] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870: Bootstrap starting.
I20260925 05:14:12.436789 22160 consensus_queue.cc:260] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.437045 22160 ts_tablet_manager.cc:1428] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:14:12.437208 21884 heartbeater.cc:503] Master 127.3.6.190:37043 was elected leader, sending a full tablet report...
I20260925 05:14:12.437750 22164 tablet_bootstrap.cc:654] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:12.438167 22164 log.cc:824] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:12.439066 22161 raft_consensus.cc:348] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.439215 22164 tablet_bootstrap.cc:492] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870: No bootstrap required, opened a new log
I20260925 05:14:12.439232 22161 raft_consensus.cc:374] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:12.439273 22164 ts_tablet_manager.cc:1397] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 05:14:12.439270 22161 raft_consensus.cc:729] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30bb68627694ea59086e7756bf5e7bb, State: Initialized, Role: FOLLOWER
I20260925 05:14:12.439409 22161 consensus_queue.cc:260] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.439669 22161 ts_tablet_manager.cc:1428] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 05:14:12.439709 22146 heartbeater.cc:503] Master 127.3.6.190:37043 was elected leader, sending a full tablet report...
I20260925 05:14:12.441295 22164 raft_consensus.cc:348] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.441423 22164 raft_consensus.cc:374] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:12.441445 22164 raft_consensus.cc:729] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f808adfb9d4708bd915703d5e12870, State: Initialized, Role: FOLLOWER
I20260925 05:14:12.441560 22164 consensus_queue.cc:260] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.441773 22164 ts_tablet_manager.cc:1428] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:14:12.441887 22015 heartbeater.cc:503] Master 127.3.6.190:37043 was elected leader, sending a full tablet report...
I20260925 05:14:12.472177 22165 raft_consensus.cc:482] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:12.472302 22165 raft_consensus.cc:504] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.472765 22165 leader_election.cc:288] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a30bb68627694ea59086e7756bf5e7bb (127.3.6.131:45901), 74f808adfb9d4708bd915703d5e12870 (127.3.6.130:39257)
I20260925 05:14:12.476115 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.473034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56213 (local address 127.3.6.130:39257)
0925 05:14:12.473079 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.473081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.473179 (+ 98us) server_negotiation.cc:407] Connection header received
0925 05:14:12.473324 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.473325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.473335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.473394 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:12.473512 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.474206 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.475296 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.475697 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.475805 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.475817 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.475999 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.476047 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:14:12.476235 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.472908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:45901 (local address 127.3.6.129:53073)
0925 05:14:12.472950 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.472962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.473014 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.473274 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.473275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.473277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:12.473367 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.473370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.474341 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.474342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.475181 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.475184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.475929 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.475930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.476016 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.476019 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.476096 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.476146 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":31}
I20260925 05:14:12.476241 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.473022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:53073 (local address 127.3.6.131:45901)
0925 05:14:12.473070 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.473072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.473083 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:14:12.473113 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.473114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.473130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.473263 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:12.473442 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.474210 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.475283 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.475820 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.475933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.475943 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.476135 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.476179 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:14:12.476138 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.472920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:39257 (local address 127.3.6.129:56213)
0925 05:14:12.473091 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.473101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:12.473154 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:12.473390 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:12.473391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:12.473393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:12.473450 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.473451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.474299 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.474301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.475197 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:12.475200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.475812 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.475813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:12.475906 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.475909 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:12.475988 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:14:12.476034 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260925 05:14:12.476689 22100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d4bc6042674e72b8f7f628d228abf0" candidate_uuid: "c4925ab9674540bca2181fc6a3080569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30bb68627694ea59086e7756bf5e7bb" is_pre_election: true
I20260925 05:14:12.476864 22100 raft_consensus.cc:2454] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4925ab9674540bca2181fc6a3080569 in term 0.
I20260925 05:14:12.476842 21969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d4bc6042674e72b8f7f628d228abf0" candidate_uuid: "c4925ab9674540bca2181fc6a3080569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f808adfb9d4708bd915703d5e12870" is_pre_election: true
I20260925 05:14:12.476989 21969 raft_consensus.cc:2454] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4925ab9674540bca2181fc6a3080569 in term 0.
I20260925 05:14:12.477156 21774 leader_election.cc:302] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30bb68627694ea59086e7756bf5e7bb, c4925ab9674540bca2181fc6a3080569; no voters:
I20260925 05:14:12.477314 22165 raft_consensus.cc:2781] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:14:12.477412 22165 raft_consensus.cc:482] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:14:12.477432 22165 raft_consensus.cc:3037] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:12.478305 22165 raft_consensus.cc:504] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.478472 22165 leader_election.cc:288] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [CANDIDATE]: Term 1 election: Requested vote from peers a30bb68627694ea59086e7756bf5e7bb (127.3.6.131:45901), 74f808adfb9d4708bd915703d5e12870 (127.3.6.130:39257)
I20260925 05:14:12.478627 22100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d4bc6042674e72b8f7f628d228abf0" candidate_uuid: "c4925ab9674540bca2181fc6a3080569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30bb68627694ea59086e7756bf5e7bb"
I20260925 05:14:12.478672 21969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d4bc6042674e72b8f7f628d228abf0" candidate_uuid: "c4925ab9674540bca2181fc6a3080569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f808adfb9d4708bd915703d5e12870"
I20260925 05:14:12.478722 22100 raft_consensus.cc:3037] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:12.478783 21969 raft_consensus.cc:3037] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:12.479468 22100 raft_consensus.cc:2454] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4925ab9674540bca2181fc6a3080569 in term 1.
I20260925 05:14:12.479676 21774 leader_election.cc:302] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30bb68627694ea59086e7756bf5e7bb, c4925ab9674540bca2181fc6a3080569; no voters:
I20260925 05:14:12.479691 21969 raft_consensus.cc:2454] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4925ab9674540bca2181fc6a3080569 in term 1.
I20260925 05:14:12.479835 22165 raft_consensus.cc:2781] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:12.480010 22165 raft_consensus.cc:686] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [term 1 LEADER]: Becoming Leader. State: Replica: c4925ab9674540bca2181fc6a3080569, State: Running, Role: LEADER
I20260925 05:14:12.480134 22165 consensus_queue.cc:237] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } }
I20260925 05:14:12.481021 21696 catalog_manager.cc:5495] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4925ab9674540bca2181fc6a3080569 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "c4925ab9674540bca2181fc6a3080569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4925ab9674540bca2181fc6a3080569" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 33245 } health_report { overall_health: HEALTHY } } }
I20260925 05:14:12.487864 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.485633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.3.6.129:33245)
0925 05:14:12.485671 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:12.485673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:12.485682 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:14:12.485745 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:12.485746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:12.485763 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:12.485826 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:12.485941 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.486416 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.487122 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:12.487302 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:12.487383 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:12.487515 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:12.487599 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:12.487669 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:12.487715 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 05:14:12.487724 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:14:12.489710 21969 raft_consensus.cc:1261] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 1 FOLLOWER]: Refusing update from remote peer c4925ab9674540bca2181fc6a3080569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:12.489717 22100 raft_consensus.cc:1261] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 1 FOLLOWER]: Refusing update from remote peer c4925ab9674540bca2181fc6a3080569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:12.490149 22170 consensus_queue.cc:1035] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a30bb68627694ea59086e7756bf5e7bb" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:12.490296 22165 consensus_queue.cc:1035] T 77d4bc6042674e72b8f7f628d228abf0 P c4925ab9674540bca2181fc6a3080569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f808adfb9d4708bd915703d5e12870" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 39257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:12.493249 22174 mvcc.cc:205] Tried to move back new op lower bound from 7333123082194984960 to 7333123082159222784. Current Snapshot: MvccSnapshot[applied={T|T < 7333123082194984960}]
I20260925 05:14:12.493887 22175 mvcc.cc:205] Tried to move back new op lower bound from 7333123082194984960 to 7333123082159222784. Current Snapshot: MvccSnapshot[applied={T|T < 7333123082194984960}]
I20260925 05:14:12.495682 21696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:12.495821 21696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:12.496654 21696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0199350292d4e66bab52d1ba1cc0e11: Sending DeleteTablet for 3 replicas of tablet 77d4bc6042674e72b8f7f628d228abf0
I20260925 05:14:12.497079 21818 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d4bc6042674e72b8f7f628d228abf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:12 UTC) from {username='slave'} at 127.0.0.1:45240
I20260925 05:14:12.497062 21949 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d4bc6042674e72b8f7f628d228abf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:12 UTC) from {username='slave'} at 127.0.0.1:46652
I20260925 05:14:12.497330 22080 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d4bc6042674e72b8f7f628d228abf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:12 UTC) from {username='slave'} at 127.0.0.1:46264
I20260925 05:14:12.497563 22184 tablet_replica.cc:330] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: stopping tablet replica
I20260925 05:14:12.497607 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 21756
I20260925 05:14:12.497673 22184 raft_consensus.cc:2229] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:12.497782 22184 raft_consensus.cc:2258] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:12.499814 22184 ts_tablet_manager.cc:1905] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:12.501711 22184 ts_tablet_manager.cc:1918] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:12.501807 22184 log.cc:1196] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/77d4bc6042674e72b8f7f628d228abf0
I20260925 05:14:12.502128 22184 ts_tablet_manager.cc:1939] T 77d4bc6042674e72b8f7f628d228abf0 P a30bb68627694ea59086e7756bf5e7bb: Deleting consensus metadata
I20260925 05:14:12.502606 21683 catalog_manager.cc:4841] TS a30bb68627694ea59086e7756bf5e7bb (127.3.6.131:45901): tablet 77d4bc6042674e72b8f7f628d228abf0 (table test-table [id=65c6dc0b64ea459b8248b9679c9b7c70]) successfully deleted
W20260925 05:14:12.503379 21681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:33245 (error 108)
I20260925 05:14:12.503968 22183 tablet_replica.cc:330] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870: stopping tablet replica
I20260925 05:14:12.503969 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 21887
I20260925 05:14:12.503988 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.503849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:33245 (local address 127.0.0.1:45264)
0925 05:14:12.503911 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.503958 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:14:12.504105 22183 raft_consensus.cc:2229] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:12.504236 22183 raft_consensus.cc:2258] T 77d4bc6042674e72b8f7f628d228abf0 P 74f808adfb9d4708bd915703d5e12870 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 05:14:12.504221 21681 catalog_manager.cc:4584] TS c4925ab9674540bca2181fc6a3080569 (127.3.6.129:33245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77d4bc6042674e72b8f7f628d228abf0: Network error: Client connection negotiation failed: client connection to 127.3.6.129:33245: connect: Connection refused (error 111)
I20260925 05:14:12.508935 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22018
I20260925 05:14:12.509001 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:12.508845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:39257 (local address 127.0.0.1:46654)
0925 05:14:12.508928 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:12.508965 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.130:39257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:14:12.513625 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 21665
2026-09-25T05:14:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (709 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:12 dist-test-slave-1lvv krb5kdc[22188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:12 dist-test-slave-1lvv krb5kdc[22188](info): set up 2 sockets
Sep 25 05:14:12 dist-test-slave-1lvv krb5kdc[22188](info): commencing operation
krb5kdc: starting...
W20260925 05:14:13.551736 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:13 dist-test-slave-1lvv krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:13.684922 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:46877
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:46877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:13.769989 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:13.770236 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:13.770262 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:13.771435 22201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:13.771485 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:13.771502 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:13.771514 22201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:13.771526 22201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:13.773011 22201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:46877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:46877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:13.773226 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:13.773422 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:13.778632 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:13.778637 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:13.778632 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:13.778858 22201 server_base.cc:1034] running on GCE node
I20260925 05:14:13.779352 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:13.779650 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:13.780797 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790313253780787 us; error 34 us; skew 500 ppm
Sep 25 05:14:13 dist-test-slave-1lvv krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313253, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:13.783396 22201 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:13.783967 22201 webserver.cc:460] Webserver started at http://127.3.6.190:41451/ using document root <none> and password file <none>
I20260925 05:14:13.784211 22201 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:13.784260 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:13.784498 22201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:13.785665 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "617b36a467234602aeefcea4edf125d0"
format_stamp: "Formatted at 2026-09-25 05:14:13 on dist-test-slave-1lvv"
server_key: "80a094bef24491838979e2a7a30327d5"
server_key_iv: "7d1729164ee15c4565c989a09d690a39"
server_key_version: "encryptionkey@0"
I20260925 05:14:13.786054 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "617b36a467234602aeefcea4edf125d0"
format_stamp: "Formatted at 2026-09-25 05:14:13 on dist-test-slave-1lvv"
server_key: "80a094bef24491838979e2a7a30327d5"
server_key_iv: "7d1729164ee15c4565c989a09d690a39"
server_key_version: "encryptionkey@0"
I20260925 05:14:13.787602 22201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:13.788451 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:13.788647 22201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:13.788748 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "617b36a467234602aeefcea4edf125d0"
format_stamp: "Formatted at 2026-09-25 05:14:13 on dist-test-slave-1lvv"
server_key: "80a094bef24491838979e2a7a30327d5"
server_key_iv: "7d1729164ee15c4565c989a09d690a39"
server_key_version: "encryptionkey@0"
I20260925 05:14:13.788828 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:13.796501 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:13.798592 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:13.798753 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:13.803282 22201 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:46877
I20260925 05:14:13.803359 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:46877 every 8 connection(s)
I20260925 05:14:13.803750 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:13.804744 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:13.807242 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0: Bootstrap starting.
I20260925 05:14:13.808015 22269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:13.808352 22269 log.cc:824] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:13.809173 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0: No bootstrap required, opened a new log
I20260925 05:14:13.810750 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22201
I20260925 05:14:13.810883 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:13.811084 3098 external_mini_cluster.cc:1428] Setting key aa8abe94d86ebba9a353c88d89290dff
I20260925 05:14:13.811229 22269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b36a467234602aeefcea4edf125d0" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46877 } }
I20260925 05:14:13.811411 22269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:13.811437 22269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617b36a467234602aeefcea4edf125d0, State: Initialized, Role: FOLLOWER
I20260925 05:14:13.811667 22269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b36a467234602aeefcea4edf125d0" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46877 } }
I20260925 05:14:13.811793 22269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:13.811846 22269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:13.811905 22269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:13.812916 22269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b36a467234602aeefcea4edf125d0" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46877 } }
I20260925 05:14:13.813056 22269 leader_election.cc:302] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617b36a467234602aeefcea4edf125d0; no voters:
I20260925 05:14:13.813256 22269 leader_election.cc:288] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:13.813313 22274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:13.813462 22274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [term 1 LEADER]: Becoming Leader. State: Replica: 617b36a467234602aeefcea4edf125d0, State: Running, Role: LEADER
I20260925 05:14:13.813548 22269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:13.813558 22274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b36a467234602aeefcea4edf125d0" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46877 } }
I20260925 05:14:13.814229 22276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617b36a467234602aeefcea4edf125d0. Latest consensus state: current_term: 1 leader_uuid: "617b36a467234602aeefcea4edf125d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b36a467234602aeefcea4edf125d0" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46877 } } }
I20260925 05:14:13.814399 22276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:13.814615 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "617b36a467234602aeefcea4edf125d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b36a467234602aeefcea4edf125d0" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46877 } } }
I20260925 05:14:13.814755 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:13.814891 22283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:13.815594 22283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:13.817435 22283 catalog_manager.cc:1347] Generated new cluster ID: a0f9172de33e47d2831b21f9f7ec3075
I20260925 05:14:13.817497 22283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:14:13 dist-test-slave-1lvv krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:13.823524 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:13.811681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.3.6.190:46877)
0925 05:14:13.811908 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:13.811925 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:13.811973 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:14:13.812189 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:13.812196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:13.812243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:13.812385 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:13.813077 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:13.814427 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:13.815557 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:13.816187 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:13.816308 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:13.819477 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:13.819483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:13.819483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:13.819497 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:13.821672 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:13.822144 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:13.822148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:13.822148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:13.822280 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:13.822514 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:13.822514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:13.822515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:13.822828 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:13.822975 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:13.823124 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 05:14:13.823263 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":126,"threads_started":1}
I20260925 05:14:13.827674 22283 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:13.828263 22283 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:13.832820 22283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0: Generated new TSK 0
I20260925 05:14:13.833053 22283 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.6.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
Entry for principal kudu/127.3.6.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab.
I20260925 05:14:13.867015 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:46877
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:13.954471 22295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:13.954757 22295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:13.954800 22295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:13.956104 22295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:13.956197 22295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:13.956244 22295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:14:13.957864 22295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.129.keytab
--principal=kudu/127.3.6.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:13.958194 22295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:13.958474 22295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:13.964012 22301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:13.964012 22300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:13.964006 22303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:13.964105 22295 server_base.cc:1034] running on GCE node
I20260925 05:14:13.964440 22295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:13.964759 22295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:13.965948 22295 hybrid_clock.cc:648] HybridClock initialized: now 1790313253965948 us; error 43 us; skew 500 ppm
Sep 25 05:14:13 dist-test-slave-1lvv krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313253, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:13.968894 22295 init.cc:377] Logged in from keytab as kudu/127.3.6.129@KRBTEST.COM (short username kudu)
I20260925 05:14:13.969372 22295 webserver.cc:460] Webserver started at http://127.3.6.129:36545/ using document root <none> and password file <none>
I20260925 05:14:13.969656 22295 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:13.969705 22295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:13.969847 22295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:13.970980 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "d3df772b77904831b816791c9037e9e4"
format_stamp: "Formatted at 2026-09-25 05:14:13 on dist-test-slave-1lvv"
server_key: "cdf09acae0483a638410eb212a326c4c"
server_key_iv: "bb208c8e28feb0bb5a9a0686a9888d15"
server_key_version: "encryptionkey@0"
I20260925 05:14:13.971374 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "d3df772b77904831b816791c9037e9e4"
format_stamp: "Formatted at 2026-09-25 05:14:13 on dist-test-slave-1lvv"
server_key: "cdf09acae0483a638410eb212a326c4c"
server_key_iv: "bb208c8e28feb0bb5a9a0686a9888d15"
server_key_version: "encryptionkey@0"
I20260925 05:14:13.973032 22295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:13.973948 22310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:13.974172 22295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:13.974288 22295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "d3df772b77904831b816791c9037e9e4"
format_stamp: "Formatted at 2026-09-25 05:14:13 on dist-test-slave-1lvv"
server_key: "cdf09acae0483a638410eb212a326c4c"
server_key_iv: "bb208c8e28feb0bb5a9a0686a9888d15"
server_key_version: "encryptionkey@0"
I20260925 05:14:13.974368 22295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:14.002192 22295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:14.004860 22295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:14.005060 22295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:14.005332 22295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:14.005681 22295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:14.005738 22295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.005774 22295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:14.005802 22295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.012199 22295 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:36409
I20260925 05:14:14.012288 22423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:36409 every 8 connection(s)
I20260925 05:14:14.012673 22295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:14:14.013531 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22295
I20260925 05:14:14.013665 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:14:14.013890 3098 external_mini_cluster.cc:1428] Setting key e7dab0e0ca621049ae3ac10b00184666
Sep 25 05:14:14 dist-test-slave-1lvv krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313253, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.129@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.6.130@KRBTEST.COM; defaulting to no policy
I20260925 05:14:14.024416 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.013593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:45129 (local address 127.3.6.190:46877)
0925 05:14:14.013654 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.013657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.013848 (+ 191us) server_negotiation.cc:407] Connection header received
0925 05:14:14.014565 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.014567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.014580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.014640 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:14.015892 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.016718 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.018218 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.018595 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.018682 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.021136 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:14.021138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:14.021138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:14.021147 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:14.022887 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.023335 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.023336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.023337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.023404 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.023681 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.023682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.023682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.023871 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:14.023952 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.024257 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.024342 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:14:14.024605 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.013545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:46877 (local address 127.3.6.129:45129)
0925 05:14:14.013710 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.013755 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.014442 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.014731 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.014735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.015490 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:14.015745 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.015751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.016840 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.016841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.018102 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.018105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.018699 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.018700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.018824 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.019440 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:14.019449 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:14.021015 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:14.022994 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:14.023001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:14.023004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:14.023248 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:14.023492 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:14.023493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:14.023493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:14.023610 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:14.023956 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:14.023964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:14.024120 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.024215 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.024358 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260925 05:14:14.025262 22424 heartbeater.cc:347] Connected to a master server at 127.3.6.190:46877
I20260925 05:14:14.025359 22424 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:14.025529 22424 heartbeater.cc:511] Master 127.3.6.190:46877 requested a full tablet report, sending...
I20260925 05:14:14.026127 22233 ts_manager.cc:194] Registered new tserver with Master: d3df772b77904831b816791c9037e9e4 (127.3.6.129:36409)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.130@KRBTEST.COM" created.
I20260925 05:14:14.026974 22233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.129@KRBTEST.COM'} at 127.3.6.129:45129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
Entry for principal kudu/127.3.6.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab.
I20260925 05:14:14.043211 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:46877
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:14.129683 22429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:14.129931 22429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:14.129959 22429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:14.131249 22429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:14.131306 22429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:14.131338 22429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:14:14.133016 22429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.130.keytab
--principal=kudu/127.3.6.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:14.133312 22429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:14.133594 22429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:14.138994 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:14.139025 22434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:14.139243 22429 server_base.cc:1034] running on GCE node
W20260925 05:14:14.138994 22437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:14.139508 22429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:14.139739 22429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:14.140954 22429 hybrid_clock.cc:648] HybridClock initialized: now 1790313254140937 us; error 32 us; skew 500 ppm
Sep 25 05:14:14 dist-test-slave-1lvv krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:14.143883 22429 init.cc:377] Logged in from keytab as kudu/127.3.6.130@KRBTEST.COM (short username kudu)
I20260925 05:14:14.144418 22429 webserver.cc:460] Webserver started at http://127.3.6.130:46565/ using document root <none> and password file <none>
I20260925 05:14:14.144686 22429 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:14.144762 22429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:14.144944 22429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:14.146147 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "eef52df4351e4343b6a33e64414fa8c8"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "cb9857bcdc5687eefbfa72972398274a"
server_key_iv: "686161d9de4c1c640a0157d3ac2a2849"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.146641 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "eef52df4351e4343b6a33e64414fa8c8"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "cb9857bcdc5687eefbfa72972398274a"
server_key_iv: "686161d9de4c1c640a0157d3ac2a2849"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.148259 22429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 05:14:14.149195 22444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.149395 22429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:14.149515 22429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "eef52df4351e4343b6a33e64414fa8c8"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "cb9857bcdc5687eefbfa72972398274a"
server_key_iv: "686161d9de4c1c640a0157d3ac2a2849"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.149601 22429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:14.158437 22429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:14.160878 22429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:14.161067 22429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:14.161350 22429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:14.161693 22429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:14.161751 22429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.161798 22429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:14.161831 22429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.167961 22429 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:37719
I20260925 05:14:14.168049 22557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:37719 every 8 connection(s)
I20260925 05:14:14.168421 22429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:14:14.169513 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22429
I20260925 05:14:14.169680 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:14:14.169883 3098 external_mini_cluster.cc:1428] Setting key e1b27d96f67cadc4d1d058bd09b20d60
Sep 25 05:14:14 dist-test-slave-1lvv krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.130@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.6.131@KRBTEST.COM; defaulting to no policy
I20260925 05:14:14.179801 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.169263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:58397 (local address 127.3.6.190:46877)
0925 05:14:14.169335 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.169337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.169619 (+ 282us) server_negotiation.cc:407] Connection header received
0925 05:14:14.170186 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.170188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.170200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.170253 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:14.171283 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.171885 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.173561 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.173793 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.173883 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.176500 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:14.176511 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:14.176512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:14.176521 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:14.178151 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.178633 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.178634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.178635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.178704 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.179001 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.179002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.179003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.179138 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:14.179228 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.179566 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.179676 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:14:14.180037 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.169248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:46877 (local address 127.3.6.130:58397)
0925 05:14:14.169456 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.169498 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.170063 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.170337 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.170340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.170959 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:14.171171 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.171175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.171991 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.171992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.173438 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.173444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.173893 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.173895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.174050 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.174664 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:14.174674 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:14.176370 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:14.178252 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:14.178264 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:14.178267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:14.178539 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:14.178778 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:14.178779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:14.178779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:14.178920 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:14.179253 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:14.179260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:14.179418 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.179545 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.179730 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":126,"threads_started":1}
I20260925 05:14:14.180619 22558 heartbeater.cc:347] Connected to a master server at 127.3.6.190:46877
I20260925 05:14:14.180740 22558 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:14.180946 22558 heartbeater.cc:511] Master 127.3.6.190:46877 requested a full tablet report, sending...
I20260925 05:14:14.181432 22233 ts_manager.cc:194] Registered new tserver with Master: eef52df4351e4343b6a33e64414fa8c8 (127.3.6.130:37719)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.131@KRBTEST.COM" created.
I20260925 05:14:14.181952 22233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.130@KRBTEST.COM'} at 127.3.6.130:58397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
Entry for principal kudu/127.3.6.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab.
I20260925 05:14:14.199851 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:46877
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:14.287431 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:14.287669 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:14.287696 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:14.289017 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:14.289078 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:14.289108 22563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:14:14.290675 22563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.131.keytab
--principal=kudu/127.3.6.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:14.290966 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:14.291210 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:14.296901 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:14.296903 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:14.296922 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:14.296970 22563 server_base.cc:1034] running on GCE node
I20260925 05:14:14.297370 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:14.297711 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:14.298878 22563 hybrid_clock.cc:648] HybridClock initialized: now 1790313254298854 us; error 43 us; skew 500 ppm
Sep 25 05:14:14 dist-test-slave-1lvv krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:14.301880 22563 init.cc:377] Logged in from keytab as kudu/127.3.6.131@KRBTEST.COM (short username kudu)
I20260925 05:14:14.302438 22563 webserver.cc:460] Webserver started at http://127.3.6.131:43389/ using document root <none> and password file <none>
I20260925 05:14:14.302736 22563 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:14.302836 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:14.302994 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:14.304203 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "2c6821806893493da8ff19d7a2e55031"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "16dd67bf7d4d2528b87a222bf012c582"
server_key_iv: "2e1bcdd910b6e229f86482dbfc99f42d"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.304544 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "2c6821806893493da8ff19d7a2e55031"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "16dd67bf7d4d2528b87a222bf012c582"
server_key_iv: "2e1bcdd910b6e229f86482dbfc99f42d"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.306051 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:14.306879 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.307142 22563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:14.307255 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "2c6821806893493da8ff19d7a2e55031"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "16dd67bf7d4d2528b87a222bf012c582"
server_key_iv: "2e1bcdd910b6e229f86482dbfc99f42d"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.307338 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:14.330842 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:14.333259 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:14.333452 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:14.333704 22563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:14.334075 22563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:14.334131 22563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.334180 22563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:14.334205 22563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.340330 22563 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:37405
I20260925 05:14:14.340375 22691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:37405 every 8 connection(s)
I20260925 05:14:14.340750 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:14:14.346046 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22563
I20260925 05:14:14.346144 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:14:14.346382 3098 external_mini_cluster.cc:1428] Setting key 3cf74d9557670f0292500801da38efa8
Sep 25 05:14:14 dist-test-slave-1lvv krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.131@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:14.350944 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.341575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:41149 (local address 127.3.6.190:46877)
0925 05:14:14.341648 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.341650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.341897 (+ 247us) server_negotiation.cc:407] Connection header received
0925 05:14:14.342430 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.342451 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.342463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.342523 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:14.343583 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.344157 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.345237 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.345428 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.345507 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.348278 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:14.348280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:14.348280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:14.348288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:14.349593 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.350068 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.350068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.350069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.350124 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.350366 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.350367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.350367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.350492 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:14.350544 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.350811 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.350877 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:14:14.351253 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.341565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:46877 (local address 127.3.6.131:41149)
0925 05:14:14.341765 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.341806 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.342295 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.342610 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.342613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.343242 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:14.343486 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.343492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.344244 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.344245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.345158 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.345160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.345533 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.345534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.345642 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.346292 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:14.346304 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:14.348188 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:14.349696 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:14.349705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:14.349707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:14.349989 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:14.350188 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:14.350189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:14.350189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:14.350305 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:14.350552 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:14.350557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:14.350691 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.350785 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.350938 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
I20260925 05:14:14.351809 22692 heartbeater.cc:347] Connected to a master server at 127.3.6.190:46877
I20260925 05:14:14.351934 22692 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:14.352140 22692 heartbeater.cc:511] Master 127.3.6.190:46877 requested a full tablet report, sending...
I20260925 05:14:14.352568 22233 ts_manager.cc:194] Registered new tserver with Master: 2c6821806893493da8ff19d7a2e55031 (127.3.6.131:37405)
I20260925 05:14:14.352982 22233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.6.131@KRBTEST.COM'} at 127.3.6.131:41149
I20260925 05:14:14.358248 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:14:14.366153 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.359838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.3.6.190:46877)
0925 05:14:14.359922 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.359924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.360059 (+ 135us) server_negotiation.cc:407] Connection header received
0925 05:14:14.360092 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.360092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.360102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.360163 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:14.360852 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.361350 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.362120 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.362384 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.362461 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.363706 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:14.363707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:14.363707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:14.363715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:14.365082 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.365436 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.365437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.365438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.365499 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:14.365677 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:14.365678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:14.365678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:14.365848 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:14.365934 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.366014 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.366083 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:14:14.368083 22233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:14:14.369767 22233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:14:14.377326 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.372415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:37719 (local address 127.0.0.1:45338)
0925 05:14:14.372814 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.372823 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.372862 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.373318 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.373319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.373321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:14.373380 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.373383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.374642 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.374643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.375883 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.375887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.376961 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.376962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.377061 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.377077 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.377149 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.377219 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":92,"thread_start_us":102,"threads_started":1}
I20260925 05:14:14.377485 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.372412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:37405 (local address 127.0.0.1:59050)
0925 05:14:14.372581 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.372631 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.372704 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.373143 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.373147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.373149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:14.373226 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.373228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.374588 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.374588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.376031 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.376033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.377217 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.377218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.377293 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.377297 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.377350 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.377401 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":125,"thread_start_us":45,"threads_started":1}
I20260925 05:14:14.377614 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.372925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.3.6.130:37719)
0925 05:14:14.373140 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.373144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.373154 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:14.373196 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.373198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.373214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.373294 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:14.373473 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.374534 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.375983 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.376764 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.376849 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.376918 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.377493 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.377538 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260925 05:14:14.377739 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.372690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.3.6.131:37405)
0925 05:14:14.372871 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.372876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.372888 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:14.372947 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.372950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.372969 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.373076 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:14.373301 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.374472 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.376138 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.377028 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.377109 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.377174 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.377599 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.377652 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260925 05:14:14.378196 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.372707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.3.6.129:36409)
0925 05:14:14.373434 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.373438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.373448 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:14.373485 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.373487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.373500 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.373598 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:14.373714 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.374414 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.376119 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.376801 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.376863 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.376918 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.377316 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.377374 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":93,"threads_started":1}
I20260925 05:14:14.378446 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.372416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36409 (local address 127.0.0.1:38518)
0925 05:14:14.372883 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.372894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.372950 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.373597 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.373598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.373599 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:14.373645 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.373647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.374531 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.374532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.376026 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.376030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.376964 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.376964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.377061 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.377077 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.377210 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.377248 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":57,"thread_start_us":103,"threads_started":1}
I20260925 05:14:14.378494 22492 tablet_service.cc:1467] Processing CreateTablet for tablet 53b8fa7ec64a4374b2cd6ceec2865dee (DEFAULT_TABLE table=test-table [id=e7396f343c2d4629b7b52cbe875967bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:14.378468 22626 tablet_service.cc:1467] Processing CreateTablet for tablet 53b8fa7ec64a4374b2cd6ceec2865dee (DEFAULT_TABLE table=test-table [id=e7396f343c2d4629b7b52cbe875967bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:14.378878 22492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b8fa7ec64a4374b2cd6ceec2865dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:14.378878 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b8fa7ec64a4374b2cd6ceec2865dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:14.379218 22358 tablet_service.cc:1467] Processing CreateTablet for tablet 53b8fa7ec64a4374b2cd6ceec2865dee (DEFAULT_TABLE table=test-table [id=e7396f343c2d4629b7b52cbe875967bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:14.379563 22358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b8fa7ec64a4374b2cd6ceec2865dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:14.382287 22709 tablet_bootstrap.cc:492] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: Bootstrap starting.
I20260925 05:14:14.382851 22711 tablet_bootstrap.cc:492] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Bootstrap starting.
I20260925 05:14:14.383224 22709 tablet_bootstrap.cc:654] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:14.383564 22711 tablet_bootstrap.cc:654] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:14.383991 22709 log.cc:824] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:14.384215 22711 log.cc:824] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:14.384616 22710 tablet_bootstrap.cc:492] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Bootstrap starting.
I20260925 05:14:14.384953 22709 tablet_bootstrap.cc:492] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: No bootstrap required, opened a new log
I20260925 05:14:14.385004 22709 ts_tablet_manager.cc:1397] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:14:14.385033 22711 tablet_bootstrap.cc:492] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: No bootstrap required, opened a new log
I20260925 05:14:14.385100 22711 ts_tablet_manager.cc:1397] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:14:14.385355 22710 tablet_bootstrap.cc:654] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:14.385967 22710 log.cc:824] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:14.386814 22710 tablet_bootstrap.cc:492] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: No bootstrap required, opened a new log
I20260925 05:14:14.386883 22710 ts_tablet_manager.cc:1397] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:14:14.386818 22709 raft_consensus.cc:348] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.387014 22709 raft_consensus.cc:374] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:14.387051 22709 raft_consensus.cc:729] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3df772b77904831b816791c9037e9e4, State: Initialized, Role: FOLLOWER
I20260925 05:14:14.387192 22709 consensus_queue.cc:260] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.387087 22711 raft_consensus.cc:348] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.387276 22711 raft_consensus.cc:374] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:14.387315 22711 raft_consensus.cc:729] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef52df4351e4343b6a33e64414fa8c8, State: Initialized, Role: FOLLOWER
I20260925 05:14:14.387488 22709 ts_tablet_manager.cc:1428] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:14:14.387470 22711 consensus_queue.cc:260] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.387531 22424 heartbeater.cc:503] Master 127.3.6.190:46877 was elected leader, sending a full tablet report...
I20260925 05:14:14.387884 22711 ts_tablet_manager.cc:1428] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:14:14.388036 22558 heartbeater.cc:503] Master 127.3.6.190:46877 was elected leader, sending a full tablet report...
I20260925 05:14:14.389146 22710 raft_consensus.cc:348] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.389356 22710 raft_consensus.cc:374] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:14.389415 22710 raft_consensus.cc:729] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6821806893493da8ff19d7a2e55031, State: Initialized, Role: FOLLOWER
I20260925 05:14:14.389541 22710 consensus_queue.cc:260] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.389829 22710 ts_tablet_manager.cc:1428] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:14:14.389863 22692 heartbeater.cc:503] Master 127.3.6.190:46877 was elected leader, sending a full tablet report...
W20260925 05:14:14.419073 22559 tablet.cc:2369] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:14:14.513571 22425 tablet.cc:2369] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:14:14.591459 22693 tablet.cc:2369] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:14:14.768258 22717 raft_consensus.cc:482] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:14.768399 22717 raft_consensus.cc:504] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.768875 22717 leader_election.cc:288] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef52df4351e4343b6a33e64414fa8c8 (127.3.6.130:37719), d3df772b77904831b816791c9037e9e4 (127.3.6.129:36409)
I20260925 05:14:14.771869 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.769163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:38091 (local address 127.3.6.130:37719)
0925 05:14:14.769217 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.769220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.769279 (+ 59us) server_negotiation.cc:407] Connection header received
0925 05:14:14.769340 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.769341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.769353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.769419 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:14.769617 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.770098 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.771100 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.771468 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.771562 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.771573 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.771727 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.771799 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:14:14.771869 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.769122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:37719 (local address 127.3.6.131:38091)
0925 05:14:14.769194 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.769209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.769273 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.769446 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.769447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.769450 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:14.769528 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.769531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.770205 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.770206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.771005 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.771007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.771552 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.771553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.771638 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.771641 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.771718 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.771792 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 05:14:14.772146 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.769133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36409 (local address 127.3.6.131:52871)
0925 05:14:14.769503 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.769512 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:14.769585 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:14.769738 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:14.769739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:14.769741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:14.769822 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.769824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.770489 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.770490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.771224 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:14.771226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.771804 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.771805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:14.771885 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.771889 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:14.771973 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 05:14:14.772022 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":200,"thread_start_us":115,"threads_started":1}
I20260925 05:14:14.772264 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.769243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:52871 (local address 127.3.6.129:36409)
0925 05:14:14.769314 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.769316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.769579 (+ 263us) server_negotiation.cc:407] Connection header received
0925 05:14:14.769644 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.769646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.769658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.769734 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:14.769898 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.770381 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.771325 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.771696 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.771803 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.771813 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.772135 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.772185 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:14:14.772245 22512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b8fa7ec64a4374b2cd6ceec2865dee" candidate_uuid: "2c6821806893493da8ff19d7a2e55031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef52df4351e4343b6a33e64414fa8c8" is_pre_election: true
I20260925 05:14:14.772430 22512 raft_consensus.cc:2454] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6821806893493da8ff19d7a2e55031 in term 0.
I20260925 05:14:14.772722 22582 leader_election.cc:302] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6821806893493da8ff19d7a2e55031, eef52df4351e4343b6a33e64414fa8c8; no voters:
I20260925 05:14:14.772853 22717 raft_consensus.cc:2781] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:14:14.772912 22717 raft_consensus.cc:482] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:14:14.772871 22378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b8fa7ec64a4374b2cd6ceec2865dee" candidate_uuid: "2c6821806893493da8ff19d7a2e55031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3df772b77904831b816791c9037e9e4" is_pre_election: true
I20260925 05:14:14.772967 22717 raft_consensus.cc:3037] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:14.773031 22378 raft_consensus.cc:2454] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6821806893493da8ff19d7a2e55031 in term 0.
I20260925 05:14:14.773773 22717 raft_consensus.cc:504] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.773926 22717 leader_election.cc:288] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [CANDIDATE]: Term 1 election: Requested vote from peers eef52df4351e4343b6a33e64414fa8c8 (127.3.6.130:37719), d3df772b77904831b816791c9037e9e4 (127.3.6.129:36409)
I20260925 05:14:14.774121 22512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b8fa7ec64a4374b2cd6ceec2865dee" candidate_uuid: "2c6821806893493da8ff19d7a2e55031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef52df4351e4343b6a33e64414fa8c8"
I20260925 05:14:14.774134 22378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b8fa7ec64a4374b2cd6ceec2865dee" candidate_uuid: "2c6821806893493da8ff19d7a2e55031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3df772b77904831b816791c9037e9e4"
I20260925 05:14:14.774207 22512 raft_consensus.cc:3037] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:14.774228 22378 raft_consensus.cc:3037] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:14.775014 22378 raft_consensus.cc:2454] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6821806893493da8ff19d7a2e55031 in term 1.
I20260925 05:14:14.775048 22512 raft_consensus.cc:2454] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6821806893493da8ff19d7a2e55031 in term 1.
I20260925 05:14:14.775254 22580 leader_election.cc:302] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6821806893493da8ff19d7a2e55031, d3df772b77904831b816791c9037e9e4; no voters:
I20260925 05:14:14.775424 22717 raft_consensus.cc:2781] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:14.775648 22717 raft_consensus.cc:686] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6821806893493da8ff19d7a2e55031, State: Running, Role: LEADER
I20260925 05:14:14.775794 22717 consensus_queue.cc:237] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } }
I20260925 05:14:14.776787 22233 catalog_manager.cc:5495] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c6821806893493da8ff19d7a2e55031 (127.3.6.131). New cstate: current_term: 1 leader_uuid: "2c6821806893493da8ff19d7a2e55031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6821806893493da8ff19d7a2e55031" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 37405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 } health_report { overall_health: UNKNOWN } } }
I20260925 05:14:14.820607 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.818066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.3.6.131:37405)
0925 05:14:14.818131 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:14.818133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:14.818144 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:14:14.818193 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:14.818194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:14.818214 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:14.818295 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:14.818451 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.818927 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.819725 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:14.819979 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:14.820063 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:14.820230 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:14.820340 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:14.820435 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:14.820462 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 05:14:14.820488 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:14:14.822726 22378 raft_consensus.cc:1261] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2c6821806893493da8ff19d7a2e55031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:14.822750 22512 raft_consensus.cc:1261] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 1 FOLLOWER]: Refusing update from remote peer 2c6821806893493da8ff19d7a2e55031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:14.823186 22719 consensus_queue.cc:1035] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef52df4351e4343b6a33e64414fa8c8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:14.823376 22717 consensus_queue.cc:1035] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3df772b77904831b816791c9037e9e4" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:14.826141 22722 mvcc.cc:205] Tried to move back new op lower bound from 7333123091750297600 to 7333123091562348544. Current Snapshot: MvccSnapshot[applied={T|T < 7333123091750297600}]
I20260925 05:14:14.826118 22724 mvcc.cc:205] Tried to move back new op lower bound from 7333123091750297600 to 7333123091562348544. Current Snapshot: MvccSnapshot[applied={T|T < 7333123091750297600}]
I20260925 05:14:14.828881 22232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:14.829021 22232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:14.830011 22232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 617b36a467234602aeefcea4edf125d0: Sending DeleteTablet for 3 replicas of tablet 53b8fa7ec64a4374b2cd6ceec2865dee
I20260925 05:14:14.830500 22626 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b8fa7ec64a4374b2cd6ceec2865dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:59050
I20260925 05:14:14.830560 22492 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b8fa7ec64a4374b2cd6ceec2865dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:45338
I20260925 05:14:14.830582 22358 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b8fa7ec64a4374b2cd6ceec2865dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:14 UTC) from {username='kudu', principal='kudu/127.3.6.190@KRBTEST.COM'} at 127.0.0.1:38518
I20260925 05:14:14.830986 22733 tablet_replica.cc:330] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4: stopping tablet replica
I20260925 05:14:14.831130 22733 raft_consensus.cc:2229] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:14.831141 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22295
I20260925 05:14:14.831316 22733 raft_consensus.cc:2258] T 53b8fa7ec64a4374b2cd6ceec2865dee P d3df772b77904831b816791c9037e9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:14.835948 22734 tablet_replica.cc:330] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: stopping tablet replica
I20260925 05:14:14.836156 22734 raft_consensus.cc:2229] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:14.836318 22732 tablet_replica.cc:330] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: stopping tablet replica
I20260925 05:14:14.836392 22734 raft_consensus.cc:2258] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:14.836493 22732 raft_consensus.cc:2229] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:14:14.836711 22732 raft_consensus.cc:2258] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:14.837200 22732 ts_tablet_manager.cc:1905] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:14.837198 22734 ts_tablet_manager.cc:1905] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 05:14:14.838642 22218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:36409 (error 108)
I20260925 05:14:14.839462 22734 ts_tablet_manager.cc:1918] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:14.839497 22732 ts_tablet_manager.cc:1918] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:14.839677 22734 log.cc:1196] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/53b8fa7ec64a4374b2cd6ceec2865dee
I20260925 05:14:14.839720 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22429
I20260925 05:14:14.839814 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.839547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:36409 (local address 127.0.0.1:38532)
0925 05:14:14.839607 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.839670 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 05:14:14.840001 22732 log.cc:1196] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/53b8fa7ec64a4374b2cd6ceec2865dee
I20260925 05:14:14.840201 22734 ts_tablet_manager.cc:1939] T 53b8fa7ec64a4374b2cd6ceec2865dee P eef52df4351e4343b6a33e64414fa8c8: Deleting consensus metadata
I20260925 05:14:14.840375 22732 ts_tablet_manager.cc:1939] T 53b8fa7ec64a4374b2cd6ceec2865dee P 2c6821806893493da8ff19d7a2e55031: Deleting consensus metadata
I20260925 05:14:14.840909 22217 catalog_manager.cc:4841] TS 2c6821806893493da8ff19d7a2e55031 (127.3.6.131:37405): tablet 53b8fa7ec64a4374b2cd6ceec2865dee (table test-table [id=e7396f343c2d4629b7b52cbe875967bc]) successfully deleted
W20260925 05:14:14.844609 22218 catalog_manager.cc:4584] TS d3df772b77904831b816791c9037e9e4 (127.3.6.129:36409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53b8fa7ec64a4374b2cd6ceec2865dee: Network error: Client connection negotiation failed: client connection to 127.3.6.129:36409: connect: Connection refused (error 111)
I20260925 05:14:14.847268 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:14.847124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:37719 (local address 127.0.0.1:45340)
0925 05:14:14.847191 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:14.847230 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.130:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 05:14:14.847294 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22563
I20260925 05:14:14.852936 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22201
2026-09-25T05:14:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2339 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T05:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:14Z Disabled control of system clock
I20260925 05:14:14.883419 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33921
--webserver_interface=127.3.6.190
--webserver_port=0
--builtin_ntp_servers=127.3.6.148:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:33921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:14:14.969133 22739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:14.969410 22739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:14.969434 22739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:14.970718 22739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:14.970783 22739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:14.970796 22739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:14.970809 22739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:14.970822 22739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:14.972409 22739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:33921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:33921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:14.972711 22739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:14.973086 22739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:14.978847 22745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:14.978847 22747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:14.978866 22744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:14.979208 22739 server_base.cc:1034] running on GCE node
I20260925 05:14:14.979499 22739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:14.979921 22739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:14.981110 22739 hybrid_clock.cc:648] HybridClock initialized: now 1790313254981092 us; error 32 us; skew 500 ppm
I20260925 05:14:14.982744 22739 webserver.cc:460] Webserver started at http://127.3.6.190:38441/ using document root <none> and password file <none>
I20260925 05:14:14.983083 22739 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:14.983165 22739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:14.983348 22739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:14.984709 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "874e7d60f4214d8dac322a869fba407c"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "735ad0be9a6a88adb058e893c11ecf34"
server_key_iv: "00a34761d60034fd9a86fa52f55f141d"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.985121 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "874e7d60f4214d8dac322a869fba407c"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "735ad0be9a6a88adb058e893c11ecf34"
server_key_iv: "00a34761d60034fd9a86fa52f55f141d"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.986706 22739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:14:14.987730 22753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:14.988023 22739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:14.988143 22739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "874e7d60f4214d8dac322a869fba407c"
format_stamp: "Formatted at 2026-09-25 05:14:14 on dist-test-slave-1lvv"
server_key: "735ad0be9a6a88adb058e893c11ecf34"
server_key_iv: "00a34761d60034fd9a86fa52f55f141d"
server_key_version: "encryptionkey@0"
I20260925 05:14:14.988236 22739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:15.004087 22739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:15.004477 22739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:15.004667 22739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:15.009152 22739 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:33921
I20260925 05:14:15.009191 22805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:33921 every 8 connection(s)
I20260925 05:14:15.009608 22739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:15.010687 22806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:15.013221 22806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c: Bootstrap starting.
I20260925 05:14:15.013804 22806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:15.014068 22806 log.cc:824] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:15.014765 22806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c: No bootstrap required, opened a new log
I20260925 05:14:15.016503 22806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e7d60f4214d8dac322a869fba407c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33921 } }
I20260925 05:14:15.016664 22806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:15.016687 22806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874e7d60f4214d8dac322a869fba407c, State: Initialized, Role: FOLLOWER
I20260925 05:14:15.016799 22806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e7d60f4214d8dac322a869fba407c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33921 } }
I20260925 05:14:15.016872 22806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:15.016894 22806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:15.016971 22806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:15.017604 22806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e7d60f4214d8dac322a869fba407c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33921 } }
I20260925 05:14:15.017798 22806 leader_election.cc:302] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874e7d60f4214d8dac322a869fba407c; no voters:
I20260925 05:14:15.018016 22806 leader_election.cc:288] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:15.018080 22809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:15.018296 22809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [term 1 LEADER]: Becoming Leader. State: Replica: 874e7d60f4214d8dac322a869fba407c, State: Running, Role: LEADER
I20260925 05:14:15.018344 22806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:15.018412 22809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e7d60f4214d8dac322a869fba407c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33921 } }
I20260925 05:14:15.018857 22811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874e7d60f4214d8dac322a869fba407c. Latest consensus state: current_term: 1 leader_uuid: "874e7d60f4214d8dac322a869fba407c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e7d60f4214d8dac322a869fba407c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33921 } } }
I20260925 05:14:15.018954 22811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:15.018944 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "874e7d60f4214d8dac322a869fba407c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e7d60f4214d8dac322a869fba407c" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 33921 } } }
I20260925 05:14:15.019032 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:15.019218 22818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:15.019328 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22739
I20260925 05:14:15.019441 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:15.019660 3098 external_mini_cluster.cc:1428] Setting key 5970fa94b040a2879a72c2b9eb34e51e
I20260925 05:14:15.019840 22818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:15.021795 22818 catalog_manager.cc:1347] Generated new cluster ID: 626bb626e23b4045893f6cfda3514799
I20260925 05:14:15.021858 22818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:15.026866 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.020372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.3.6.190:33921)
0925 05:14:15.020779 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.020790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.020821 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:14:15.020945 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.020952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.021037 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.021180 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:15.021550 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.022924 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.024135 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.024859 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.024988 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.025820 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:15.025824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:15.025825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:15.026074 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:15.026121 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:15.026137 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:15.026220 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.026347 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.026439 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":65,"threads_started":1}
I20260925 05:14:15.036700 22818 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:15.037312 22818 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:15.051961 22818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c: Generated new TSK 0
I20260925 05:14:15.052234 22818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:15.055045 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.129:0
--local_ip_for_outbound_sockets=127.3.6.129
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33921
--builtin_ntp_servers=127.3.6.148:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:14:15.168416 22830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:15.168690 22830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:15.168730 22830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:15.169973 22830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:15.170046 22830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:15.170091 22830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.129
I20260925 05:14:15.171530 22830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.6.129
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.129
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:15.171875 22830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:15.172211 22830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:15.178380 22835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:15.178437 22836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:15.178359 22838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:15.178543 22830 server_base.cc:1034] running on GCE node
I20260925 05:14:15.178936 22830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:15.179229 22830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:15.180444 22830 hybrid_clock.cc:648] HybridClock initialized: now 1790313255180419 us; error 40 us; skew 500 ppm
I20260925 05:14:15.182024 22830 webserver.cc:460] Webserver started at http://127.3.6.129:46627/ using document root <none> and password file <none>
I20260925 05:14:15.182305 22830 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:15.182381 22830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:15.182539 22830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:15.183802 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data/instance:
uuid: "499faec7714c4cc180d4df8dd22c1e06"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "b8c358bfc75712e7957926da73505f4d"
server_key_iv: "2ab10c794f23a4a5733a0bad9079a466"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.184219 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance:
uuid: "499faec7714c4cc180d4df8dd22c1e06"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "b8c358bfc75712e7957926da73505f4d"
server_key_iv: "2ab10c794f23a4a5733a0bad9079a466"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.185743 22830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:15.186553 22844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.186733 22830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:15.186849 22830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/wal
uuid: "499faec7714c4cc180d4df8dd22c1e06"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "b8c358bfc75712e7957926da73505f4d"
server_key_iv: "2ab10c794f23a4a5733a0bad9079a466"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.186930 22830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:15.206146 22830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:15.206573 22830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:15.206765 22830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:15.207054 22830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:15.207489 22830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:15.207549 22830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.207585 22830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:15.207617 22830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.217005 22830 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.129:44753
I20260925 05:14:15.217058 22957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.129:44753 every 8 connection(s)
I20260925 05:14:15.217523 22830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/data/info.pb
I20260925 05:14:15.222926 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22830
I20260925 05:14:15.223052 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-0/wal/instance
I20260925 05:14:15.223249 3098 external_mini_cluster.cc:1428] Setting key 92e97295ed7d38cdbf530cf0597a7567
I20260925 05:14:15.224043 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.218627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:56597 (local address 127.3.6.190:33921)
0925 05:14:15.218757 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.218760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.219169 (+ 409us) server_negotiation.cc:407] Connection header received
0925 05:14:15.219842 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.219843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.219862 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.219990 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:15.220794 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.221375 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.222411 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.222628 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.222735 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.223710 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:15.223712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:15.223712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:15.223718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:15.223736 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:15.223740 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:15.223845 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.223927 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.223956 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 05:14:15.224308 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.218589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33921 (local address 127.3.6.129:56597)
0925 05:14:15.218919 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.218982 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.219676 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.220081 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.220085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.220095 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:15.220687 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.220692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.221488 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.221488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.222323 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.222325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.222745 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.222746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.222862 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.223509 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:15.223510 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:15.223541 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:15.223595 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:15.223849 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:15.223854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:15.223857 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.223917 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.224047 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":140,"threads_started":1}
I20260925 05:14:15.224517 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.130:0
--local_ip_for_outbound_sockets=127.3.6.130
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33921
--builtin_ntp_servers=127.3.6.148:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:14:15.226609 22958 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33921
I20260925 05:14:15.226799 22958 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:15.227115 22958 heartbeater.cc:511] Master 127.3.6.190:33921 requested a full tablet report, sending...
I20260925 05:14:15.228072 22770 ts_manager.cc:194] Registered new tserver with Master: 499faec7714c4cc180d4df8dd22c1e06 (127.3.6.129:44753)
I20260925 05:14:15.228991 22770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.129:56597
W20260925 05:14:15.317173 22961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:15.317445 22961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:15.317485 22961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:15.318840 22961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:15.318905 22961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:15.318951 22961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.130
I20260925 05:14:15.320588 22961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.6.130
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.130
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:15.320871 22961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:15.321204 22961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:15.326646 22969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:15.326611 22966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:15.326606 22967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:15.326960 22961 server_base.cc:1034] running on GCE node
I20260925 05:14:15.327248 22961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:15.327544 22961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:15.328718 22961 hybrid_clock.cc:648] HybridClock initialized: now 1790313255328709 us; error 38 us; skew 500 ppm
I20260925 05:14:15.330195 22961 webserver.cc:460] Webserver started at http://127.3.6.130:33877/ using document root <none> and password file <none>
I20260925 05:14:15.330531 22961 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:15.330611 22961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:15.330776 22961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:15.332039 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data/instance:
uuid: "c1c36bf58b984a639ae944c6a5bb4bd8"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "7cd5d3951f87fe5e7cfc8104c08b5496"
server_key_iv: "e5555d6caf89cb9b8c56a351628658be"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.332415 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance:
uuid: "c1c36bf58b984a639ae944c6a5bb4bd8"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "7cd5d3951f87fe5e7cfc8104c08b5496"
server_key_iv: "e5555d6caf89cb9b8c56a351628658be"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.333849 22961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:15.334602 22975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.334780 22961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:15.334890 22961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal
uuid: "c1c36bf58b984a639ae944c6a5bb4bd8"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "7cd5d3951f87fe5e7cfc8104c08b5496"
server_key_iv: "e5555d6caf89cb9b8c56a351628658be"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.334971 22961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:15.348321 22961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:15.348685 22961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:15.348857 22961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:15.349110 22961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:15.349493 22961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:15.349550 22961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.349589 22961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:15.349637 22961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.356365 22961 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.130:34251
I20260925 05:14:15.356422 23088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.130:34251 every 8 connection(s)
I20260925 05:14:15.356902 22961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/data/info.pb
I20260925 05:14:15.361088 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 22961
I20260925 05:14:15.361265 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal/instance
I20260925 05:14:15.361512 3098 external_mini_cluster.cc:1428] Setting key 56fff9bf35add47456d6ab2eeaa17ebc
I20260925 05:14:15.363060 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.6.131:0
--local_ip_for_outbound_sockets=127.3.6.131
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33921
--builtin_ntp_servers=127.3.6.148:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:14:15.364755 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.357925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.130:44445 (local address 127.3.6.190:33921)
0925 05:14:15.358109 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.358120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.358339 (+ 219us) server_negotiation.cc:407] Connection header received
0925 05:14:15.359026 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.359027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.359041 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.359130 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:15.360189 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.360842 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.362606 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.363062 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.363186 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.364416 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:15.364418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:15.364418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:15.364424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:15.364443 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:15.364448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:15.364511 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.364594 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.364629 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260925 05:14:15.364872 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.357956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33921 (local address 127.3.6.130:44445)
0925 05:14:15.358185 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.358245 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.358898 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.359209 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.359215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.359225 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:15.360054 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.360065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.360988 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.360989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.362476 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.362479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.363224 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.363226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.363521 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.364258 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:15.364258 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:15.364290 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:15.364315 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:15.364517 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:15.364522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:15.364524 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.364581 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.364667 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260925 05:14:15.365391 23089 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33921
I20260925 05:14:15.365513 23089 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:15.365737 23089 heartbeater.cc:511] Master 127.3.6.190:33921 requested a full tablet report, sending...
I20260925 05:14:15.366129 22770 ts_manager.cc:194] Registered new tserver with Master: c1c36bf58b984a639ae944c6a5bb4bd8 (127.3.6.130:34251)
I20260925 05:14:15.366516 22770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.130:44445
W20260925 05:14:15.451699 23092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:15.452154 23092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:15.452212 23092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:15.453946 23092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:15.454054 23092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:15.454110 23092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.6.131
I20260925 05:14:15.456163 23092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.6.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.6.131
--webserver_port=0
--tserver_master_addrs=127.3.6.190:33921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.6.131
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:15.456578 23092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:15.457059 23092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:15.464344 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:15.464327 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:15.464326 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:15.464926 23092 server_base.cc:1034] running on GCE node
I20260925 05:14:15.465148 23092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:15.465405 23092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:15.466609 23092 hybrid_clock.cc:648] HybridClock initialized: now 1790313255466606 us; error 48 us; skew 500 ppm
I20260925 05:14:15.468062 23092 webserver.cc:460] Webserver started at http://127.3.6.131:45743/ using document root <none> and password file <none>
I20260925 05:14:15.468360 23092 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:15.468437 23092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:15.468601 23092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:15.469784 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data/instance:
uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "bc2fa7ed9d80848e8c7ab343efcfe2a6"
server_key_iv: "fd14f0c278c9cd271c780a361e78ebf1"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.470184 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance:
uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "bc2fa7ed9d80848e8c7ab343efcfe2a6"
server_key_iv: "fd14f0c278c9cd271c780a361e78ebf1"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.471685 23092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:15.472599 23106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.472792 23092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:15.472905 23092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal
uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c"
format_stamp: "Formatted at 2026-09-25 05:14:15 on dist-test-slave-1lvv"
server_key: "bc2fa7ed9d80848e8c7ab343efcfe2a6"
server_key_iv: "fd14f0c278c9cd271c780a361e78ebf1"
server_key_version: "encryptionkey@0"
I20260925 05:14:15.472986 23092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:15.481238 23092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:15.481638 23092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:15.481814 23092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:15.482095 23092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:14:15.482527 23092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:14:15.482594 23092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.482640 23092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:14:15.482672 23092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:15.490077 23092 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.131:33863
I20260925 05:14:15.490227 23219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.131:33863 every 8 connection(s)
I20260925 05:14:15.490600 23092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/data/info.pb
I20260925 05:14:15.496867 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.491468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.131:41515 (local address 127.3.6.190:33921)
0925 05:14:15.491541 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.491543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.491801 (+ 258us) server_negotiation.cc:407] Connection header received
0925 05:14:15.492548 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.492550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.492566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.492620 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:15.493400 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.493947 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.495086 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.495302 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.495407 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.496570 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:15.496572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:15.496572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:15.496578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:15.496598 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:15.496603 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:15.496664 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.496756 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.496794 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:14:15.497120 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.491403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:33921 (local address 127.3.6.131:41515)
0925 05:14:15.491652 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.491690 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.492405 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.492707 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.492711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.492720 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:15.493270 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.493275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.494085 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.494086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.494974 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.494976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.495425 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.495426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.495589 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.496385 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:14:15.496386 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:15.496422 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:14:15.496449 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:15.496680 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:15.496685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:15.496688 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.496746 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.496846 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260925 05:14:15.497731 23220 heartbeater.cc:347] Connected to a master server at 127.3.6.190:33921
I20260925 05:14:15.497824 23220 heartbeater.cc:464] Registering TS with master...
I20260925 05:14:15.498055 23220 heartbeater.cc:511] Master 127.3.6.190:33921 requested a full tablet report, sending...
I20260925 05:14:15.498519 22770 ts_manager.cc:194] Registered new tserver with Master: 91f93b0ea94c4e5e8b1a5a8866ccbf7c (127.3.6.131:33863)
I20260925 05:14:15.498936 22770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.6.131:41515
I20260925 05:14:15.500140 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23092
I20260925 05:14:15.500236 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal/instance
I20260925 05:14:15.500381 3098 external_mini_cluster.cc:1428] Setting key 96058dc7b7aaaea4a6509969c5e5c88c
I20260925 05:14:15.501825 3098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:14:15.506654 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.503339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.3.6.190:33921)
0925 05:14:15.503507 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.503509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.503572 (+ 63us) server_negotiation.cc:407] Connection header received
0925 05:14:15.503611 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.503612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.503624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.503685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:15.503907 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.504411 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.505216 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.505395 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.505448 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.506210 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:15.506211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:15.506211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:15.506216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:15.506231 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:15.506235 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:15.506287 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.506440 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.506511 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260925 05:14:15.508725 22770 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:14:15.510360 22770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:14:15.517645 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.512828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:44753 (local address 127.0.0.1:41208)
0925 05:14:15.513655 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.513663 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.513709 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.513959 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.513960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.513961 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:15.514000 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.514001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.514890 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.514891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.516275 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.516278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.517291 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.517292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.517381 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.517399 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.517490 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.517548 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":26,"thread_start_us":108,"threads_started":1}
I20260925 05:14:15.517656 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.513100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.3.6.129:44753)
0925 05:14:15.513354 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.513358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.513769 (+ 411us) server_negotiation.cc:407] Connection header received
0925 05:14:15.513813 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.513815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.513830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.513895 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:15.514087 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.514794 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.516397 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.517187 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.517278 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.517382 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.517508 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.517563 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":115,"threads_started":1}
I20260925 05:14:15.517742 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.513053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.3.6.131:33863)
0925 05:14:15.513256 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.513262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.513582 (+ 320us) server_negotiation.cc:407] Connection header received
0925 05:14:15.513652 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.513654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.513675 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.513749 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:15.513962 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.515143 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.516404 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.517187 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.517292 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.517382 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.517602 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.517643 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260925 05:14:15.517982 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.512823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:33863 (local address 127.0.0.1:42002)
0925 05:14:15.513510 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.513517 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.513579 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.513825 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.513826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.513828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:15.513890 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.513892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.515226 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.515227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.516319 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.516321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.517417 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.517419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.517503 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.517507 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.517818 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.517874 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260925 05:14:15.518582 22891 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec97a3efddb4db2bcead27ad67e2997 (DEFAULT_TABLE table=test-table [id=0fec9aef4f8f4cbfa5ba7f232ad470b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:15.519073 22891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec97a3efddb4db2bcead27ad67e2997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:15.519090 23154 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec97a3efddb4db2bcead27ad67e2997 (DEFAULT_TABLE table=test-table [id=0fec9aef4f8f4cbfa5ba7f232ad470b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:15.519328 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.512823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:34251 (local address 127.0.0.1:59634)
0925 05:14:15.513250 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.513302 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.513400 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.513731 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.513735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.513737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:15.513815 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.513818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.516417 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.516417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.517080 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.517082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.519047 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.519048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.519124 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.519129 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.519211 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.519259 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":103,"threads_started":1}
I20260925 05:14:15.519479 23154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec97a3efddb4db2bcead27ad67e2997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:15.519590 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.513053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.3.6.130:34251)
0925 05:14:15.513257 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.513262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.513435 (+ 173us) server_negotiation.cc:407] Connection header received
0925 05:14:15.513528 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.513530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.513547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.513640 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:15.513900 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.515134 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.517175 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.518929 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.519018 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.519115 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.519439 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.519507 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260925 05:14:15.520531 23023 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec97a3efddb4db2bcead27ad67e2997 (DEFAULT_TABLE table=test-table [id=0fec9aef4f8f4cbfa5ba7f232ad470b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:14:15.520886 23023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec97a3efddb4db2bcead27ad67e2997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:15.523262 23235 tablet_bootstrap.cc:492] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Bootstrap starting.
I20260925 05:14:15.524045 23235 tablet_bootstrap.cc:654] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:15.524386 23236 tablet_bootstrap.cc:492] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Bootstrap starting.
I20260925 05:14:15.524434 23235 log.cc:824] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:15.524501 23234 tablet_bootstrap.cc:492] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: Bootstrap starting.
I20260925 05:14:15.525205 23234 tablet_bootstrap.cc:654] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:15.525207 23236 tablet_bootstrap.cc:654] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:15.525523 23234 log.cc:824] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:15.525568 23235 tablet_bootstrap.cc:492] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: No bootstrap required, opened a new log
I20260925 05:14:15.525550 23236 log.cc:824] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:15.525628 23235 ts_tablet_manager.cc:1397] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:14:15.526258 23234 tablet_bootstrap.cc:492] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: No bootstrap required, opened a new log
I20260925 05:14:15.526353 23234 ts_tablet_manager.cc:1397] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:14:15.527886 23235 raft_consensus.cc:348] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.528053 23235 raft_consensus.cc:374] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:15.528079 23235 raft_consensus.cc:729] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f93b0ea94c4e5e8b1a5a8866ccbf7c, State: Initialized, Role: FOLLOWER
I20260925 05:14:15.528223 23235 consensus_queue.cc:260] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.528522 23235 ts_tablet_manager.cc:1428] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 05:14:15.528607 23220 heartbeater.cc:503] Master 127.3.6.190:33921 was elected leader, sending a full tablet report...
I20260925 05:14:15.528540 23234 raft_consensus.cc:348] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.528713 23234 raft_consensus.cc:374] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:15.528760 23234 raft_consensus.cc:729] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499faec7714c4cc180d4df8dd22c1e06, State: Initialized, Role: FOLLOWER
I20260925 05:14:15.528870 23234 consensus_queue.cc:260] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.529178 23234 ts_tablet_manager.cc:1428] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:14:15.529337 22958 heartbeater.cc:503] Master 127.3.6.190:33921 was elected leader, sending a full tablet report...
I20260925 05:14:15.529790 23236 tablet_bootstrap.cc:492] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: No bootstrap required, opened a new log
I20260925 05:14:15.529884 23236 ts_tablet_manager.cc:1397] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 05:14:15.532073 23236 raft_consensus.cc:348] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.532223 23236 raft_consensus.cc:374] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:15.532245 23236 raft_consensus.cc:729] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c36bf58b984a639ae944c6a5bb4bd8, State: Initialized, Role: FOLLOWER
I20260925 05:14:15.532366 23236 consensus_queue.cc:260] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.532620 23236 ts_tablet_manager.cc:1428] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:14:15.532747 23089 heartbeater.cc:503] Master 127.3.6.190:33921 was elected leader, sending a full tablet report...
W20260925 05:14:15.607542 23090 tablet.cc:2369] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:14:15.718519 22959 tablet.cc:2369] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:14:15.741340 23221 tablet.cc:2369] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:14:15.841588 23241 raft_consensus.cc:482] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:14:15.841763 23241 raft_consensus.cc:504] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.842201 23241 leader_election.cc:288] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91f93b0ea94c4e5e8b1a5a8866ccbf7c (127.3.6.131:33863), c1c36bf58b984a639ae944c6a5bb4bd8 (127.3.6.130:34251)
I20260925 05:14:15.845515 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.842583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:47907 (local address 127.3.6.130:34251)
0925 05:14:15.842627 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.842629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.842678 (+ 49us) server_negotiation.cc:407] Connection header received
0925 05:14:15.842801 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.842802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.842811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.842901 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:15.843100 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.843565 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.844720 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.845100 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.845159 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.845192 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.845391 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.845433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:14:15.845518 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.842561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.6.129:44603 (local address 127.3.6.131:33863)
0925 05:14:15.842614 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.842616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.842704 (+ 88us) server_negotiation.cc:407] Connection header received
0925 05:14:15.842800 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.842801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.842811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.842864 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:14:15.843067 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.843564 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.844720 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.845110 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.845207 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.845217 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.845389 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.845434 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 05:14:15.845558 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.842479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.131:33863 (local address 127.3.6.129:44603)
0925 05:14:15.842621 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.842632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.842703 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.842897 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.842899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.842901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:15.842988 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.842990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.843661 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.843663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.844604 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.844607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.845229 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.845230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.845306 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.845309 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.845382 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.845437 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":50,"thread_start_us":74,"threads_started":1}
I20260925 05:14:15.845561 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.842435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.130:34251 (local address 127.3.6.129:47907)
0925 05:14:15.842609 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.842622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:15.842713 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:15.842925 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:15.842926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:15.842928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:14:15.843005 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.843007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.843661 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.843663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.844612 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:15.844615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.845202 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.845203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:15.845295 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.845308 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:14:15.845382 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:14:15.845431 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260925 05:14:15.845966 23174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec97a3efddb4db2bcead27ad67e2997" candidate_uuid: "499faec7714c4cc180d4df8dd22c1e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" is_pre_election: true
I20260925 05:14:15.846172 23174 raft_consensus.cc:2454] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 499faec7714c4cc180d4df8dd22c1e06 in term 0.
I20260925 05:14:15.846150 23043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec97a3efddb4db2bcead27ad67e2997" candidate_uuid: "499faec7714c4cc180d4df8dd22c1e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" is_pre_election: true
I20260925 05:14:15.846326 23043 raft_consensus.cc:2454] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 499faec7714c4cc180d4df8dd22c1e06 in term 0.
I20260925 05:14:15.846494 22847 leader_election.cc:302] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499faec7714c4cc180d4df8dd22c1e06, 91f93b0ea94c4e5e8b1a5a8866ccbf7c; no voters:
I20260925 05:14:15.846653 23241 raft_consensus.cc:2781] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:14:15.846716 23241 raft_consensus.cc:482] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:14:15.846735 23241 raft_consensus.cc:3037] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:15.847491 23241 raft_consensus.cc:504] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.847656 23241 leader_election.cc:288] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [CANDIDATE]: Term 1 election: Requested vote from peers 91f93b0ea94c4e5e8b1a5a8866ccbf7c (127.3.6.131:33863), c1c36bf58b984a639ae944c6a5bb4bd8 (127.3.6.130:34251)
I20260925 05:14:15.847836 23174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec97a3efddb4db2bcead27ad67e2997" candidate_uuid: "499faec7714c4cc180d4df8dd22c1e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c"
I20260925 05:14:15.847894 23043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec97a3efddb4db2bcead27ad67e2997" candidate_uuid: "499faec7714c4cc180d4df8dd22c1e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8"
I20260925 05:14:15.847937 23174 raft_consensus.cc:3037] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:15.847980 23043 raft_consensus.cc:3037] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:15.848696 23043 raft_consensus.cc:2454] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 499faec7714c4cc180d4df8dd22c1e06 in term 1.
I20260925 05:14:15.848794 23174 raft_consensus.cc:2454] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 499faec7714c4cc180d4df8dd22c1e06 in term 1.
I20260925 05:14:15.848881 22846 leader_election.cc:302] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499faec7714c4cc180d4df8dd22c1e06, c1c36bf58b984a639ae944c6a5bb4bd8; no voters:
I20260925 05:14:15.848987 23241 raft_consensus.cc:2781] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:15.849136 23241 raft_consensus.cc:686] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 1 LEADER]: Becoming Leader. State: Replica: 499faec7714c4cc180d4df8dd22c1e06, State: Running, Role: LEADER
I20260925 05:14:15.849247 23241 consensus_queue.cc:237] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } }
I20260925 05:14:15.850116 22770 catalog_manager.cc:5495] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 499faec7714c4cc180d4df8dd22c1e06 (127.3.6.129). New cstate: current_term: 1 leader_uuid: "499faec7714c4cc180d4df8dd22c1e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "499faec7714c4cc180d4df8dd22c1e06" member_type: VOTER last_known_addr { host: "127.3.6.129" port: 44753 } health_report { overall_health: HEALTHY } } }
I20260925 05:14:15.872562 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.870069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.3.6.129:44753)
0925 05:14:15.870141 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:15.870143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:15.870177 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:14:15.870226 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:15.870227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:15.870248 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:15.870329 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:15.870467 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.870945 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.871704 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:15.871958 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:15.872047 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:15.872210 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:15.872326 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:15.872426 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:15.872445 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 05:14:15.872466 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:14:15.874554 23174 raft_consensus.cc:1261] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 1 FOLLOWER]: Refusing update from remote peer 499faec7714c4cc180d4df8dd22c1e06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:15.874573 23043 raft_consensus.cc:1261] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 1 FOLLOWER]: Refusing update from remote peer 499faec7714c4cc180d4df8dd22c1e06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:15.874953 23245 consensus_queue.cc:1035] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f93b0ea94c4e5e8b1a5a8866ccbf7c" member_type: VOTER last_known_addr { host: "127.3.6.131" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:15.875141 23241 consensus_queue.cc:1035] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1c36bf58b984a639ae944c6a5bb4bd8" member_type: VOTER last_known_addr { host: "127.3.6.130" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:15.877600 23248 mvcc.cc:205] Tried to move back new op lower bound from 7333123096059080704 to 7333123095959228416. Current Snapshot: MvccSnapshot[applied={T|T < 7333123096059080704}]
I20260925 05:14:15.878365 23249 mvcc.cc:205] Tried to move back new op lower bound from 7333123096059080704 to 7333123095959228416. Current Snapshot: MvccSnapshot[applied={T|T < 7333123096059080704}]
I20260925 05:14:15.879781 23250 mvcc.cc:205] Tried to move back new op lower bound from 7333123096059080704 to 7333123095959228416. Current Snapshot: MvccSnapshot[applied={T|T < 7333123096059080704}]
I20260925 05:14:15.880247 22769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:15.880357 22769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:14:15.881189 22769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 874e7d60f4214d8dac322a869fba407c: Sending DeleteTablet for 3 replicas of tablet 2ec97a3efddb4db2bcead27ad67e2997
I20260925 05:14:15.881577 23154 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec97a3efddb4db2bcead27ad67e2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:15 UTC) from {username='slave'} at 127.0.0.1:42002
I20260925 05:14:15.881561 23023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec97a3efddb4db2bcead27ad67e2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:15 UTC) from {username='slave'} at 127.0.0.1:59634
I20260925 05:14:15.881826 23258 tablet_replica.cc:330] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: stopping tablet replica
I20260925 05:14:15.881811 22891 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec97a3efddb4db2bcead27ad67e2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:14:15 UTC) from {username='slave'} at 127.0.0.1:41208
I20260925 05:14:15.881943 23258 raft_consensus.cc:2229] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:15.882069 23258 raft_consensus.cc:2258] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:15.882081 23260 tablet_replica.cc:330] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06: stopping tablet replica
I20260925 05:14:15.882264 23260 raft_consensus.cc:2229] T 2ec97a3efddb4db2bcead27ad67e2997 P 499faec7714c4cc180d4df8dd22c1e06 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:14:15.882331 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22830
I20260925 05:14:15.883913 23258 ts_tablet_manager.cc:1905] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:15.886099 23259 tablet_replica.cc:330] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: stopping tablet replica
I20260925 05:14:15.886228 23258 ts_tablet_manager.cc:1918] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:15.886245 23259 raft_consensus.cc:2229] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:14:15.886318 23258 log.cc:1196] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-2/wal/wals/2ec97a3efddb4db2bcead27ad67e2997
I20260925 05:14:15.886373 23259 raft_consensus.cc:2258] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:14:15.886639 23258 ts_tablet_manager.cc:1939] T 2ec97a3efddb4db2bcead27ad67e2997 P 91f93b0ea94c4e5e8b1a5a8866ccbf7c: Deleting consensus metadata
I20260925 05:14:15.886965 23259 ts_tablet_manager.cc:1905] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:14:15.887147 22756 catalog_manager.cc:4841] TS 91f93b0ea94c4e5e8b1a5a8866ccbf7c (127.3.6.131:33863): tablet 2ec97a3efddb4db2bcead27ad67e2997 (table test-table [id=0fec9aef4f8f4cbfa5ba7f232ad470b2]) successfully deleted
I20260925 05:14:15.888945 23259 ts_tablet_manager.cc:1918] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:14:15.889038 23259 log.cc:1196] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Deleting WAL directory at /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790313008371521-3098-0/minicluster-data/ts-1/wal/wals/2ec97a3efddb4db2bcead27ad67e2997
I20260925 05:14:15.889324 23259 ts_tablet_manager.cc:1939] T 2ec97a3efddb4db2bcead27ad67e2997 P c1c36bf58b984a639ae944c6a5bb4bd8: Deleting consensus metadata
W20260925 05:14:15.889415 22757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.6.129:44753 (error 108)
I20260925 05:14:15.889808 22755 catalog_manager.cc:4841] TS c1c36bf58b984a639ae944c6a5bb4bd8 (127.3.6.130:34251): tablet 2ec97a3efddb4db2bcead27ad67e2997 (table test-table [id=0fec9aef4f8f4cbfa5ba7f232ad470b2]) successfully deleted
I20260925 05:14:15.889971 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:15.889807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.129:44753 (local address 127.0.0.1:41228)
0925 05:14:15.889867 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:15.889934 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.129:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 05:14:15.890148 22757 catalog_manager.cc:4584] TS 499faec7714c4cc180d4df8dd22c1e06 (127.3.6.129:44753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ec97a3efddb4db2bcead27ad67e2997: Network error: Client connection negotiation failed: client connection to 127.3.6.129:44753: connect: Connection refused (error 111)
I20260925 05:14:15.890256 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22961
I20260925 05:14:15.896047 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23092
I20260925 05:14:15.901309 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 22739
2026-09-25T05:14:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1047 ms)
[----------] 6 tests from EncryptionPolicyTest (9058 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:15 dist-test-slave-1lvv krb5kdc[23264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:15 dist-test-slave-1lvv krb5kdc[23264](info): set up 2 sockets
Sep 25 05:14:15 dist-test-slave-1lvv krb5kdc[23264](info): commencing operation
krb5kdc: starting...
W20260925 05:14:16.948495 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:17 dist-test-slave-1lvv krb5kdc[23264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:17.086441 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:17.171412 23277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:17.171706 23277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:17.171749 23277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:17.173062 23277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:17.173149 23277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:17.173175 23277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:17.173197 23277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:17.173226 23277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:17.174839 23277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:17.175163 23277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:17.175423 23277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:17.180794 23282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:17.180796 23283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:17.180794 23285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:17.181041 23277 server_base.cc:1034] running on GCE node
I20260925 05:14:17.181331 23277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:17.181718 23277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:17.182900 23277 hybrid_clock.cc:648] HybridClock initialized: now 1790313257182872 us; error 42 us; skew 500 ppm
Sep 25 05:14:17 dist-test-slave-1lvv krb5kdc[23264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:17.189321 23277 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:17.189769 23277 webserver.cc:460] Webserver started at http://127.0.0.1:36107/ using document root <none> and password file <none>
I20260925 05:14:17.189970 23277 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:17.190016 23277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:17.190120 23277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:17.191231 23277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "85d4295987514827ab7b15ed5ccb13f0"
format_stamp: "Formatted at 2026-09-25 05:14:17 on dist-test-slave-1lvv"
server_key: "958be7b725cf309a0cf6615183f00029"
server_key_iv: "5d7ae7601875b0430e34443fdc958446"
server_key_version: "encryptionkey@0"
I20260925 05:14:17.191572 23277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "85d4295987514827ab7b15ed5ccb13f0"
format_stamp: "Formatted at 2026-09-25 05:14:17 on dist-test-slave-1lvv"
server_key: "958be7b725cf309a0cf6615183f00029"
server_key_iv: "5d7ae7601875b0430e34443fdc958446"
server_key_version: "encryptionkey@0"
I20260925 05:14:17.193209 23277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:14:17.194088 23292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:17.194305 23277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:17.194406 23277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "85d4295987514827ab7b15ed5ccb13f0"
format_stamp: "Formatted at 2026-09-25 05:14:17 on dist-test-slave-1lvv"
server_key: "958be7b725cf309a0cf6615183f00029"
server_key_iv: "5d7ae7601875b0430e34443fdc958446"
server_key_version: "encryptionkey@0"
I20260925 05:14:17.194474 23277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:17.212602 23277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:17.214736 23277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:17.214861 23277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:17.219034 23277 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46073
I20260925 05:14:17.219116 23344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46073 every 8 connection(s)
I20260925 05:14:17.219491 23277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:17.220554 23345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:17.222570 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23277
I20260925 05:14:17.222694 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:17.222885 3098 external_mini_cluster.cc:1428] Setting key bfa1cd9d0fe51ab026dc4b7ba9da2a03
I20260925 05:14:17.223413 23345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0: Bootstrap starting.
I20260925 05:14:17.224145 23345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:17.224457 23345 log.cc:824] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:17.225225 23345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0: No bootstrap required, opened a new log
I20260925 05:14:17.227161 23345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4295987514827ab7b15ed5ccb13f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260925 05:14:17.227355 23345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:17.227406 23345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d4295987514827ab7b15ed5ccb13f0, State: Initialized, Role: FOLLOWER
I20260925 05:14:17.227586 23345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4295987514827ab7b15ed5ccb13f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260925 05:14:17.227694 23345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:17.227756 23345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:17.227840 23345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:17.228657 23345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4295987514827ab7b15ed5ccb13f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260925 05:14:17.228792 23345 leader_election.cc:302] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85d4295987514827ab7b15ed5ccb13f0; no voters:
I20260925 05:14:17.229024 23345 leader_election.cc:288] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:17.229056 23350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:17.229192 23350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [term 1 LEADER]: Becoming Leader. State: Replica: 85d4295987514827ab7b15ed5ccb13f0, State: Running, Role: LEADER
I20260925 05:14:17.229336 23345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:17.229354 23350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4295987514827ab7b15ed5ccb13f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260925 05:14:17.229780 23350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85d4295987514827ab7b15ed5ccb13f0. Latest consensus state: current_term: 1 leader_uuid: "85d4295987514827ab7b15ed5ccb13f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4295987514827ab7b15ed5ccb13f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } } }
I20260925 05:14:17.229895 23351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85d4295987514827ab7b15ed5ccb13f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4295987514827ab7b15ed5ccb13f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } } }
I20260925 05:14:17.230003 23351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:17.229930 23350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0 [sys.catalog]: This master's current role is: LEADER
Sep 25 05:14:17 dist-test-slave-1lvv krb5kdc[23264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260925 05:14:17.231446 23366 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:17.231577 23366 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:17.231666 23357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:17.231875 23357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:17.233500 23357 catalog_manager.cc:1347] Generated new cluster ID: 2dafaf4f9f7148bc8c05321aeb46fe51
I20260925 05:14:17.233556 23357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:17.235261 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:17.223588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42801 (local address 127.0.0.1:46073)
0925 05:14:17.223792 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:17.223800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:17.223827 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:14:17.224315 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:17.224325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:17.224391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:17.224590 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:17.225281 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:17.226229 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:17.227057 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:17.227596 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:17.227726 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:17.230885 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:17.230889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:17.230890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:17.230904 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:17.233011 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:17.233518 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:17.233521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:17.233521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:17.233630 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:17.233896 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:17.233896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:17.233897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:17.234222 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:17.234386 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:17.234907 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 05:14:17.235026 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260925 05:14:17.236207 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:17.245620 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:17.237755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44431 (local address 127.0.0.1:46073)
0925 05:14:17.237803 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:17.237806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:17.238002 (+ 196us) server_negotiation.cc:407] Connection header received
0925 05:14:17.238035 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:17.238035 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:17.238045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:17.238112 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:17.238722 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:17.239520 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:17.240554 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:17.240914 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:17.241001 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:17.242517 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:17.242518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:17.242518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:17.242526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:17.244289 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:17.244736 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:17.244737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:17.244737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:17.244805 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:17.245042 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:17.245043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:17.245044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:17.245188 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:17.245316 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:17.245440 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 05:14:17.245513 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:14:17.261686 23357 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:17.262457 23357 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:17.269466 23357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85d4295987514827ab7b15ed5ccb13f0: Generated new TSK 0
I20260925 05:14:17.269698 23357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:17.273177 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23277
W20260925 05:14:17.278208 23262 connection.cc:535] client connection to 127.0.0.1:46073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46073): Connection reset by peer (error 104)
2026-09-25T05:14:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:17 dist-test-slave-1lvv krb5kdc[23376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:17 dist-test-slave-1lvv krb5kdc[23376](info): set up 2 sockets
Sep 25 05:14:17 dist-test-slave-1lvv krb5kdc[23376](info): commencing operation
krb5kdc: starting...
W20260925 05:14:18.316262 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:18 dist-test-slave-1lvv krb5kdc[23376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:18.447439 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:18.532335 23389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:18.532608 23389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:18.532653 23389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:18.533883 23389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:18.533972 23389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:18.534000 23389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:18.534034 23389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:18.534060 23389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:18.535629 23389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:18.535951 23389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:18.536216 23389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:18.541313 23397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:18.541380 23394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:18.541313 23395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:18.541486 23389 server_base.cc:1034] running on GCE node
I20260925 05:14:18.541791 23389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:18.542094 23389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:18.543241 23389 hybrid_clock.cc:648] HybridClock initialized: now 1790313258543227 us; error 48 us; skew 500 ppm
Sep 25 05:14:18 dist-test-slave-1lvv krb5kdc[23376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:18.546300 23389 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:18.546844 23389 webserver.cc:460] Webserver started at http://127.0.0.1:43079/ using document root <none> and password file <none>
I20260925 05:14:18.547134 23389 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:18.547211 23389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:18.547374 23389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:18.548583 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "239ae0a220224054a2e980b042c463da"
format_stamp: "Formatted at 2026-09-25 05:14:18 on dist-test-slave-1lvv"
server_key: "b620ae09aaa4d837cd17b0c87765e87f"
server_key_iv: "34a0a38bc47e180fb797e278b3258572"
server_key_version: "encryptionkey@0"
I20260925 05:14:18.548975 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "239ae0a220224054a2e980b042c463da"
format_stamp: "Formatted at 2026-09-25 05:14:18 on dist-test-slave-1lvv"
server_key: "b620ae09aaa4d837cd17b0c87765e87f"
server_key_iv: "34a0a38bc47e180fb797e278b3258572"
server_key_version: "encryptionkey@0"
I20260925 05:14:18.550381 23389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:14:18.551136 23404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:18.551306 23389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:18.551417 23389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "239ae0a220224054a2e980b042c463da"
format_stamp: "Formatted at 2026-09-25 05:14:18 on dist-test-slave-1lvv"
server_key: "b620ae09aaa4d837cd17b0c87765e87f"
server_key_iv: "34a0a38bc47e180fb797e278b3258572"
server_key_version: "encryptionkey@0"
I20260925 05:14:18.551498 23389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:18.570317 23389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:18.572738 23389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:18.572921 23389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:18.577476 23389 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45625
I20260925 05:14:18.577510 23456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45625 every 8 connection(s)
I20260925 05:14:18.577912 23389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:18.579013 23457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:18.581804 23457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da: Bootstrap starting.
I20260925 05:14:18.582492 23457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:18.582799 23457 log.cc:824] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:18.583564 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23389
I20260925 05:14:18.583693 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:18.583752 23457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da: No bootstrap required, opened a new log
I20260925 05:14:18.583956 3098 external_mini_cluster.cc:1428] Setting key 9c0a8423808ef21de73d9ae25d4fc255
I20260925 05:14:18.585773 23457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239ae0a220224054a2e980b042c463da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260925 05:14:18.585968 23457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:18.586042 23457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239ae0a220224054a2e980b042c463da, State: Initialized, Role: FOLLOWER
I20260925 05:14:18.586232 23457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239ae0a220224054a2e980b042c463da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260925 05:14:18.586334 23457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:18.586386 23457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:18.586442 23457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:18.587091 23457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239ae0a220224054a2e980b042c463da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260925 05:14:18.587239 23457 leader_election.cc:302] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 239ae0a220224054a2e980b042c463da; no voters:
I20260925 05:14:18.587451 23457 leader_election.cc:288] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:18.587482 23462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:18.587607 23462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [term 1 LEADER]: Becoming Leader. State: Replica: 239ae0a220224054a2e980b042c463da, State: Running, Role: LEADER
I20260925 05:14:18.587779 23457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:18.587790 23462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239ae0a220224054a2e980b042c463da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } }
I20260925 05:14:18.588315 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 239ae0a220224054a2e980b042c463da. Latest consensus state: current_term: 1 leader_uuid: "239ae0a220224054a2e980b042c463da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239ae0a220224054a2e980b042c463da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } } }
I20260925 05:14:18.588436 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:18.588667 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "239ae0a220224054a2e980b042c463da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "239ae0a220224054a2e980b042c463da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45625 } } }
I20260925 05:14:18.588980 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:18.589205 23470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:18.590107 23470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:18.591857 23470 catalog_manager.cc:1347] Generated new cluster ID: e028e7edf4bd48adaa9a4e45199a6fe9
I20260925 05:14:18.591926 23470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:18.592396 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:18.584551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33141 (local address 127.0.0.1:45625)
0925 05:14:18.584771 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:18.584783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:18.584817 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:14:18.585750 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:18.585756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:18.585806 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:18.585962 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:18.586250 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:18.587695 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:18.588859 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:18.589507 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:18.589652 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:18.591487 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:18.591490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:18.591491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:18.591750 (+ 259us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:18.591819 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:18.591837 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:18.591947 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:18.592129 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:14:18.592164 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260925 05:14:18.596937 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:18.601730 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:18.598506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53559 (local address 127.0.0.1:45625)
0925 05:14:18.598551 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:18.598552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:18.598682 (+ 130us) server_negotiation.cc:407] Connection header received
0925 05:14:18.598764 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:18.598765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:18.598777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:18.598835 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:18.598999 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:18.599499 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:18.600346 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:18.600594 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:18.600676 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:18.601415 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:18.601417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:18.601417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:18.601421 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:18.601437 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:18.601442 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:18.601540 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:18.601606 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 05:14:18.601619 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:14:18.605182 23470 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:18.605907 23470 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:18.621996 23470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 239ae0a220224054a2e980b042c463da: Generated new TSK 0
I20260925 05:14:18.622267 23470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:18.628540 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23389
W20260925 05:14:18.634744 23374 connection.cc:535] client connection to 127.0.0.1:45625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45625): Connection reset by peer (error 104)
2026-09-25T05:14:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:18 dist-test-slave-1lvv krb5kdc[23488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:18 dist-test-slave-1lvv krb5kdc[23488](info): set up 2 sockets
Sep 25 05:14:18 dist-test-slave-1lvv krb5kdc[23488](info): commencing operation
krb5kdc: starting...
W20260925 05:14:19.676069 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:19 dist-test-slave-1lvv krb5kdc[23488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:19.804963 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:19.890695 23501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:19.890985 23501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:19.891031 23501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:19.892397 23501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:19.892491 23501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:19.892514 23501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:19.892537 23501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:19.892571 23501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:19.894430 23501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:19.894793 23501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:19.895061 23501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:19.900434 23506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:19.900439 23507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:19.900576 23501 server_base.cc:1034] running on GCE node
W20260925 05:14:19.900439 23509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:19.900996 23501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:19.901276 23501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:19.902537 23501 hybrid_clock.cc:648] HybridClock initialized: now 1790313259902532 us; error 36 us; skew 500 ppm
Sep 25 05:14:19 dist-test-slave-1lvv krb5kdc[23488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:19.905197 23501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:19.905705 23501 webserver.cc:460] Webserver started at http://127.0.0.1:32793/ using document root <none> and password file <none>
I20260925 05:14:19.905969 23501 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:19.906046 23501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:19.906199 23501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:19.907347 23501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "d3bbd3fc5c17496f903e51129cc412db"
format_stamp: "Formatted at 2026-09-25 05:14:19 on dist-test-slave-1lvv"
server_key: "352d87740b26e968d3888e827bb32667"
server_key_iv: "8eb70e977b6b94622697bb00b484eae6"
server_key_version: "encryptionkey@0"
I20260925 05:14:19.907797 23501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "d3bbd3fc5c17496f903e51129cc412db"
format_stamp: "Formatted at 2026-09-25 05:14:19 on dist-test-slave-1lvv"
server_key: "352d87740b26e968d3888e827bb32667"
server_key_iv: "8eb70e977b6b94622697bb00b484eae6"
server_key_version: "encryptionkey@0"
I20260925 05:14:19.909268 23501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:19.910110 23516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:19.910349 23501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:19.910457 23501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "d3bbd3fc5c17496f903e51129cc412db"
format_stamp: "Formatted at 2026-09-25 05:14:19 on dist-test-slave-1lvv"
server_key: "352d87740b26e968d3888e827bb32667"
server_key_iv: "8eb70e977b6b94622697bb00b484eae6"
server_key_version: "encryptionkey@0"
I20260925 05:14:19.910578 23501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:19.914265 23501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:19.916447 23501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:19.916636 23501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:19.920905 23501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46193
I20260925 05:14:19.920943 23568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46193 every 8 connection(s)
I20260925 05:14:19.921327 23501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:19.922439 23569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:19.925017 23569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db: Bootstrap starting.
I20260925 05:14:19.925678 23569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:19.925987 23569 log.cc:824] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:19.926815 23569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db: No bootstrap required, opened a new log
I20260925 05:14:19.928639 23569 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bbd3fc5c17496f903e51129cc412db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46193 } }
I20260925 05:14:19.928802 23569 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:19.928826 23569 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bbd3fc5c17496f903e51129cc412db, State: Initialized, Role: FOLLOWER
I20260925 05:14:19.929036 23569 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bbd3fc5c17496f903e51129cc412db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46193 } }
I20260925 05:14:19.929126 23569 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:19.929171 23569 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:19.929224 23569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:19.929939 23569 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bbd3fc5c17496f903e51129cc412db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46193 } }
I20260925 05:14:19.930063 23569 leader_election.cc:302] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3bbd3fc5c17496f903e51129cc412db; no voters:
I20260925 05:14:19.930260 23569 leader_election.cc:288] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:19.930351 23572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:19.930532 23572 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [term 1 LEADER]: Becoming Leader. State: Replica: d3bbd3fc5c17496f903e51129cc412db, State: Running, Role: LEADER
I20260925 05:14:19.930704 23572 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bbd3fc5c17496f903e51129cc412db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46193 } }
I20260925 05:14:19.930804 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23501
I20260925 05:14:19.930917 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:19.930552 23569 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:19.931133 3098 external_mini_cluster.cc:1428] Setting key 1f07ad5e210cc342f9a2a4a851990c4d
I20260925 05:14:19.931555 23573 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3bbd3fc5c17496f903e51129cc412db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bbd3fc5c17496f903e51129cc412db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46193 } } }
I20260925 05:14:19.931700 23573 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:19.932077 23583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:19.932111 23574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bbd3fc5c17496f903e51129cc412db. Latest consensus state: current_term: 1 leader_uuid: "d3bbd3fc5c17496f903e51129cc412db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bbd3fc5c17496f903e51129cc412db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46193 } } }
I20260925 05:14:19.932261 23574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:19.932767 23583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:19.934119 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:19.931891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33623 (local address 127.0.0.1:46193)
0925 05:14:19.932404 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:19.932415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:19.932452 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:14:19.932592 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:19.932599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:19.932632 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:19.932799 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:19.933557 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:19.933561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:19.933562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:19.933566 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:19.933615 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:19.933632 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:19.933689 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:19.933873 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 05:14:19.933919 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":81,"threads_started":1}
I20260925 05:14:19.934676 23583 catalog_manager.cc:1347] Generated new cluster ID: 38a4a0e745f64def9d0b7bc75160f046
I20260925 05:14:19.934744 23583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:19.935247 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:19.941147 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:19.936725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47987 (local address 127.0.0.1:46193)
0925 05:14:19.936769 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:19.936771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:19.939998 (+ 3227us) server_negotiation.cc:407] Connection header received
0925 05:14:19.940069 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:19.940070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:19.940079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:19.940154 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:19.940877 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:19.940880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:19.940880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:19.940885 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:19.940904 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:19.940909 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:19.940986 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:19.941037 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 05:14:19.941050 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:14:19.963657 23583 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:19.963930 23583 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:19.973218 23583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3bbd3fc5c17496f903e51129cc412db: Generated new TSK 0
I20260925 05:14:19.973474 23583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:19.977116 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23501
2026-09-25T05:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:20 dist-test-slave-1lvv krb5kdc[23600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:20 dist-test-slave-1lvv krb5kdc[23600](info): set up 2 sockets
Sep 25 05:14:20 dist-test-slave-1lvv krb5kdc[23600](info): commencing operation
krb5kdc: starting...
W20260925 05:14:21.016055 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:21 dist-test-slave-1lvv krb5kdc[23600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:21.152459 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:21.237767 23613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:21.238055 23613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:21.238099 23613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:21.239307 23613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:21.239391 23613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:21.239414 23613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:21.239435 23613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:21.239466 23613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:21.241104 23613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:21.241436 23613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:21.241698 23613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:21.247267 23621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:21.247255 23619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:21.247258 23618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:21.247340 23613 server_base.cc:1034] running on GCE node
I20260925 05:14:21.247859 23613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:21.248201 23613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:21.249378 23613 hybrid_clock.cc:648] HybridClock initialized: now 1790313261249366 us; error 34 us; skew 500 ppm
Sep 25 05:14:21 dist-test-slave-1lvv krb5kdc[23600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:21.252156 23613 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:21.252696 23613 webserver.cc:460] Webserver started at http://127.0.0.1:42461/ using document root <none> and password file <none>
I20260925 05:14:21.252961 23613 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:21.253036 23613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:21.253186 23613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:21.254328 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "4cb50c4c816d4a04be77bd25e52db26f"
format_stamp: "Formatted at 2026-09-25 05:14:21 on dist-test-slave-1lvv"
server_key: "c29260fb146bec8e423ede391265736a"
server_key_iv: "ca282886dcc37da0059cd9a1c42484f5"
server_key_version: "encryptionkey@0"
I20260925 05:14:21.254721 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "4cb50c4c816d4a04be77bd25e52db26f"
format_stamp: "Formatted at 2026-09-25 05:14:21 on dist-test-slave-1lvv"
server_key: "c29260fb146bec8e423ede391265736a"
server_key_iv: "ca282886dcc37da0059cd9a1c42484f5"
server_key_version: "encryptionkey@0"
I20260925 05:14:21.256209 23613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:21.257072 23628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:21.257311 23613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:21.257437 23613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "4cb50c4c816d4a04be77bd25e52db26f"
format_stamp: "Formatted at 2026-09-25 05:14:21 on dist-test-slave-1lvv"
server_key: "c29260fb146bec8e423ede391265736a"
server_key_iv: "ca282886dcc37da0059cd9a1c42484f5"
server_key_version: "encryptionkey@0"
I20260925 05:14:21.257521 23613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:21.273693 23613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:21.275998 23613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:21.276185 23613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:21.280588 23613 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42357
I20260925 05:14:21.280632 23680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42357 every 8 connection(s)
I20260925 05:14:21.281032 23613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:21.282090 23681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:21.284579 23681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f: Bootstrap starting.
I20260925 05:14:21.285140 23681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:21.285387 23681 log.cc:824] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:21.286123 23681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f: No bootstrap required, opened a new log
I20260925 05:14:21.287791 23681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb50c4c816d4a04be77bd25e52db26f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260925 05:14:21.287946 23681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:21.287971 23681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb50c4c816d4a04be77bd25e52db26f, State: Initialized, Role: FOLLOWER
I20260925 05:14:21.288082 23681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb50c4c816d4a04be77bd25e52db26f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260925 05:14:21.288136 23681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:21.288158 23681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:21.288240 23681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:21.288698 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23613
I20260925 05:14:21.288834 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:21.289060 3098 external_mini_cluster.cc:1428] Setting key e8b84ad13e41c6a46814f413384f5940
I20260925 05:14:21.289050 23681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb50c4c816d4a04be77bd25e52db26f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260925 05:14:21.289163 23681 leader_election.cc:302] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cb50c4c816d4a04be77bd25e52db26f; no voters:
I20260925 05:14:21.289387 23681 leader_election.cc:288] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:21.289417 23684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:21.289548 23684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [term 1 LEADER]: Becoming Leader. State: Replica: 4cb50c4c816d4a04be77bd25e52db26f, State: Running, Role: LEADER
I20260925 05:14:21.289625 23684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb50c4c816d4a04be77bd25e52db26f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260925 05:14:21.289669 23681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:21.291158 23686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cb50c4c816d4a04be77bd25e52db26f. Latest consensus state: current_term: 1 leader_uuid: "4cb50c4c816d4a04be77bd25e52db26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb50c4c816d4a04be77bd25e52db26f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } } }
I20260925 05:14:21.291356 23686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:21.291525 23685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cb50c4c816d4a04be77bd25e52db26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb50c4c816d4a04be77bd25e52db26f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } } }
I20260925 05:14:21.291787 23685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:21.291846 23702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:21.292385 23702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:21.294034 23702 catalog_manager.cc:1347] Generated new cluster ID: 3aef7aa34bc24f9fb3abe647adefe7b1
I20260925 05:14:21.294096 23702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:14:21 dist-test-slave-1lvv krb5kdc[23600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:14:21.299674 23702 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:21.300333 23702 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:21.301877 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:21.289531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42221 (local address 127.0.0.1:42357)
0925 05:14:21.289762 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:21.289771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:21.290049 (+ 278us) server_negotiation.cc:407] Connection header received
0925 05:14:21.290161 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:21.290166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:21.290211 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:21.290351 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:21.291292 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:21.292735 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:21.293865 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:21.294476 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:21.294587 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:21.297692 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:21.297697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:21.297698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:21.297713 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:21.300067 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:21.300537 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:21.300540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:21.300541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:21.300668 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:21.300898 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:21.300899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:21.300900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:21.301228 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:21.301416 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:21.301489 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 05:14:21.301655 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260925 05:14:21.302942 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:21.306489 23702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cb50c4c816d4a04be77bd25e52db26f: Generated new TSK 0
I20260925 05:14:21.306708 23702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:21.311940 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:21.304665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51841 (local address 127.0.0.1:42357)
0925 05:14:21.304700 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:21.304702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:21.304710 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:21.304773 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:21.304774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:21.304783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:21.304837 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:21.305530 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:21.306241 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:21.307403 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:21.307616 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:21.307671 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:21.309167 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:21.309168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:21.309168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:21.309175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:21.310701 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:21.311141 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:21.311141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:21.311142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:21.311196 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:21.311402 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:21.311403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:21.311403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:21.311510 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:21.311584 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:21.311725 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 05:14:21.311871 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:14:21.314563 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23613
2026-09-25T05:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:21 dist-test-slave-1lvv krb5kdc[23712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:21 dist-test-slave-1lvv krb5kdc[23712](info): set up 2 sockets
Sep 25 05:14:21 dist-test-slave-1lvv krb5kdc[23712](info): commencing operation
krb5kdc: starting...
W20260925 05:14:22.372211 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:22 dist-test-slave-1lvv krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:22.507191 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:22.597085 23725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:22.597384 23725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:22.597429 23725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:22.598677 23725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:22.598752 23725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:22.598776 23725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:22.598798 23725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:22.598827 23725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:22.600472 23725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:22.600790 23725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:22.601042 23725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:22.606097 23731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:22.606051 23733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:22.606263 23725 server_base.cc:1034] running on GCE node
W20260925 05:14:22.606065 23730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:22.606783 23725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:22.607059 23725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:22.608224 23725 hybrid_clock.cc:648] HybridClock initialized: now 1790313262608207 us; error 30 us; skew 500 ppm
Sep 25 05:14:22 dist-test-slave-1lvv krb5kdc[23712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:22.615438 23725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:22.616096 23725 webserver.cc:460] Webserver started at http://127.0.0.1:43135/ using document root <none> and password file <none>
I20260925 05:14:22.616379 23725 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:22.616430 23725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:22.616622 23725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:22.617789 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "a326c5eb22084fb09176e008cdd5a844"
format_stamp: "Formatted at 2026-09-25 05:14:22 on dist-test-slave-1lvv"
server_key: "d274a74e1f6a992e13fee4a4624baff9"
server_key_iv: "499862bcc1b611001d1ecdf5e879a995"
server_key_version: "encryptionkey@0"
I20260925 05:14:22.618206 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "a326c5eb22084fb09176e008cdd5a844"
format_stamp: "Formatted at 2026-09-25 05:14:22 on dist-test-slave-1lvv"
server_key: "d274a74e1f6a992e13fee4a4624baff9"
server_key_iv: "499862bcc1b611001d1ecdf5e879a995"
server_key_version: "encryptionkey@0"
I20260925 05:14:22.619969 23725 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 05:14:22.620930 23740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:22.621218 23725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:22.621335 23725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "a326c5eb22084fb09176e008cdd5a844"
format_stamp: "Formatted at 2026-09-25 05:14:22 on dist-test-slave-1lvv"
server_key: "d274a74e1f6a992e13fee4a4624baff9"
server_key_iv: "499862bcc1b611001d1ecdf5e879a995"
server_key_version: "encryptionkey@0"
I20260925 05:14:22.621417 23725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:22.634186 23725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:22.636886 23725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:22.637096 23725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:22.641526 23725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36513
I20260925 05:14:22.641557 23792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36513 every 8 connection(s)
I20260925 05:14:22.641987 23725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:22.643123 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:22.643405 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23725
I20260925 05:14:22.643577 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:22.643841 3098 external_mini_cluster.cc:1428] Setting key f85e8d643540b30439d4ce8e486185d3
I20260925 05:14:22.646205 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844: Bootstrap starting.
I20260925 05:14:22.646996 23793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:22.647332 23793 log.cc:824] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:22.648192 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844: No bootstrap required, opened a new log
I20260925 05:14:22.649804 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:22.644412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48663 (local address 127.0.0.1:36513)
0925 05:14:22.644627 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:22.644635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:22.644662 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:14:22.645102 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:22.645109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:22.645159 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:22.645291 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:22.645553 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:22.646476 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:22.647695 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:22.648173 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:22.648301 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:22.649049 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:22.649053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:22.649053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:22.649224 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:22.649256 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:22.649268 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:22.649347 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:22.649425 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:14:22.649494 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260925 05:14:22.650058 23793 raft_consensus.cc:348] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a326c5eb22084fb09176e008cdd5a844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260925 05:14:22.650256 23793 raft_consensus.cc:374] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:22.650296 23793 raft_consensus.cc:729] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a326c5eb22084fb09176e008cdd5a844, State: Initialized, Role: FOLLOWER
I20260925 05:14:22.650434 23793 consensus_queue.cc:260] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a326c5eb22084fb09176e008cdd5a844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260925 05:14:22.650537 23793 raft_consensus.cc:388] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:22.650594 23793 raft_consensus.cc:482] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:22.650655 23793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:22.650897 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:22.651377 23793 raft_consensus.cc:504] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a326c5eb22084fb09176e008cdd5a844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260925 05:14:22.651499 23793 leader_election.cc:302] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a326c5eb22084fb09176e008cdd5a844; no voters:
I20260925 05:14:22.651711 23793 leader_election.cc:288] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:22.651741 23798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:22.651899 23798 raft_consensus.cc:686] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [term 1 LEADER]: Becoming Leader. State: Replica: a326c5eb22084fb09176e008cdd5a844, State: Running, Role: LEADER
I20260925 05:14:22.652010 23793 sys_catalog.cc:564] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:22.652052 23798 consensus_queue.cc:237] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a326c5eb22084fb09176e008cdd5a844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260925 05:14:22.652585 23800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a326c5eb22084fb09176e008cdd5a844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a326c5eb22084fb09176e008cdd5a844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260925 05:14:22.652604 23801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a326c5eb22084fb09176e008cdd5a844. Latest consensus state: current_term: 1 leader_uuid: "a326c5eb22084fb09176e008cdd5a844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a326c5eb22084fb09176e008cdd5a844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260925 05:14:22.652690 23800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:22.652722 23801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:22.652895 23812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:22.653432 23812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:22.655321 23812 catalog_manager.cc:1347] Generated new cluster ID: a39a26427b494f1d83a93073eb7f8248
I20260925 05:14:22.655381 23812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:22.656389 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:22.652307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46971 (local address 127.0.0.1:36513)
0925 05:14:22.652339 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:22.652341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:22.652591 (+ 250us) server_negotiation.cc:407] Connection header received
0925 05:14:22.653182 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:22.653184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:22.653195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:22.653248 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:22.653438 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:22.654035 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:22.654991 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:22.655236 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:22.655301 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:22.656146 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:22.656147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:22.656147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:22.656152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:22.656168 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:22.656172 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:22.656225 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:22.656285 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 05:14:22.656310 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:14:22.675705 23812 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:22.676539 23812 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:22.681847 23812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a326c5eb22084fb09176e008cdd5a844: Generated new TSK 0
I20260925 05:14:22.682088 23812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:22.691407 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23725
2026-09-25T05:14:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:22 dist-test-slave-1lvv krb5kdc[23824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:22 dist-test-slave-1lvv krb5kdc[23824](info): set up 2 sockets
Sep 25 05:14:22 dist-test-slave-1lvv krb5kdc[23824](info): commencing operation
krb5kdc: starting...
W20260925 05:14:23.748168 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:23 dist-test-slave-1lvv krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:23.888130 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:23.984258 23837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:23.984488 23837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:23.984513 23837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:23.985685 23837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:23.985728 23837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:23.985742 23837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:23.985756 23837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:23.985769 23837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:23.987288 23837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:23.987504 23837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:23.987707 23837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:23.993647 23842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:23.993654 23845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:23.993830 23837 server_base.cc:1034] running on GCE node
W20260925 05:14:23.993654 23843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:23.994218 23837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:23.994520 23837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:23.995667 23837 hybrid_clock.cc:648] HybridClock initialized: now 1790313263995657 us; error 31 us; skew 500 ppm
Sep 25 05:14:23 dist-test-slave-1lvv krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:23.998301 23837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:23.998831 23837 webserver.cc:460] Webserver started at http://127.0.0.1:42495/ using document root <none> and password file <none>
I20260925 05:14:23.999097 23837 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:23.999151 23837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:23.999332 23837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:24.000556 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "65ba3810ab2446a7a2d618770bca62e8"
format_stamp: "Formatted at 2026-09-25 05:14:23 on dist-test-slave-1lvv"
server_key: "1c97792bd94dc3cc33f80b44a4c470f6"
server_key_iv: "acfbfbef4ae17824241296249e57b016"
server_key_version: "encryptionkey@0"
I20260925 05:14:24.000957 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "65ba3810ab2446a7a2d618770bca62e8"
format_stamp: "Formatted at 2026-09-25 05:14:23 on dist-test-slave-1lvv"
server_key: "1c97792bd94dc3cc33f80b44a4c470f6"
server_key_iv: "acfbfbef4ae17824241296249e57b016"
server_key_version: "encryptionkey@0"
I20260925 05:14:24.002391 23837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:24.003224 23852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:24.003487 23837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:24.003612 23837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "65ba3810ab2446a7a2d618770bca62e8"
format_stamp: "Formatted at 2026-09-25 05:14:23 on dist-test-slave-1lvv"
server_key: "1c97792bd94dc3cc33f80b44a4c470f6"
server_key_iv: "acfbfbef4ae17824241296249e57b016"
server_key_version: "encryptionkey@0"
I20260925 05:14:24.003695 23837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:24.007285 23837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:24.009649 23837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:24.009857 23837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:24.014298 23837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38077
I20260925 05:14:24.014353 23904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38077 every 8 connection(s)
I20260925 05:14:24.014731 23837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:24.015748 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:24.018230 23905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8: Bootstrap starting.
I20260925 05:14:24.018791 23905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:24.019052 23905 log.cc:824] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:24.019845 23905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8: No bootstrap required, opened a new log
I20260925 05:14:24.021435 23905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ba3810ab2446a7a2d618770bca62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260925 05:14:24.021592 23905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:24.021618 23905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ba3810ab2446a7a2d618770bca62e8, State: Initialized, Role: FOLLOWER
I20260925 05:14:24.021726 23905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ba3810ab2446a7a2d618770bca62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260925 05:14:24.021780 23905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:24.021801 23905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:24.021836 23905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:24.022361 23905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ba3810ab2446a7a2d618770bca62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260925 05:14:24.022449 23905 leader_election.cc:302] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ba3810ab2446a7a2d618770bca62e8; no voters:
I20260925 05:14:24.022580 23905 leader_election.cc:288] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:24.022666 23908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:24.022871 23905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:24.022907 23908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [term 1 LEADER]: Becoming Leader. State: Replica: 65ba3810ab2446a7a2d618770bca62e8, State: Running, Role: LEADER
I20260925 05:14:24.023075 23908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ba3810ab2446a7a2d618770bca62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260925 05:14:24.023790 23910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ba3810ab2446a7a2d618770bca62e8. Latest consensus state: current_term: 1 leader_uuid: "65ba3810ab2446a7a2d618770bca62e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ba3810ab2446a7a2d618770bca62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260925 05:14:24.023895 23910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:24.023967 23909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ba3810ab2446a7a2d618770bca62e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ba3810ab2446a7a2d618770bca62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260925 05:14:24.024087 23909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:24.024197 23918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:24.024331 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23837
I20260925 05:14:24.024451 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:24.024683 3098 external_mini_cluster.cc:1428] Setting key 36bd5301f367e9e619d2216e8eee5adc
I20260925 05:14:24.024816 23918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:24.026561 23918 catalog_manager.cc:1347] Generated new cluster ID: 43cb332e8f484454a0f2af06a8fc1122
I20260925 05:14:24.026620 23918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:24.027058 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:24.025246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37663 (local address 127.0.0.1:38077)
0925 05:14:24.025439 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:24.025449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:24.025482 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:14:24.025601 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:24.025609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:24.025643 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:24.025789 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:24.026525 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:24.026528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:24.026529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:24.026534 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:24.026579 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:24.026590 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:24.026684 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:24.026817 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 05:14:24.026865 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260925 05:14:24.027915 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:24.030480 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:24.029310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55801 (local address 127.0.0.1:38077)
0925 05:14:24.029346 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:24.029348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:24.029491 (+ 143us) server_negotiation.cc:407] Connection header received
0925 05:14:24.029571 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:24.029572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:24.029577 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:24.029622 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:24.030231 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:24.030232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:24.030233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:24.030238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:24.030252 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:24.030257 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:24.030314 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:24.030371 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 05:14:24.030387 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:14:24.048921 23918 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:24.049160 23918 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:24.055465 23918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ba3810ab2446a7a2d618770bca62e8: Generated new TSK 0
I20260925 05:14:24.055697 23918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:24.060988 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23837
2026-09-25T05:14:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:24 dist-test-slave-1lvv krb5kdc[23936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:24 dist-test-slave-1lvv krb5kdc[23936](info): set up 2 sockets
Sep 25 05:14:24 dist-test-slave-1lvv krb5kdc[23936](info): commencing operation
krb5kdc: starting...
W20260925 05:14:25.116568 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:25 dist-test-slave-1lvv krb5kdc[23936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:25.246591 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:25.329483 23949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:25.329737 23949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:25.329779 23949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:25.330947 23949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:25.331013 23949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:25.331041 23949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:25.331073 23949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:25.331097 23949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:25.333307 23949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:25.333739 23949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:25.334112 23949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:25.339522 23957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:25.339522 23955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:25.339555 23954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:25.339691 23949 server_base.cc:1034] running on GCE node
I20260925 05:14:25.340078 23949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:25.340418 23949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:25.341552 23949 hybrid_clock.cc:648] HybridClock initialized: now 1790313265341544 us; error 38 us; skew 500 ppm
Sep 25 05:14:25 dist-test-slave-1lvv krb5kdc[23936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:25.344573 23949 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:25.345042 23949 webserver.cc:460] Webserver started at http://127.0.0.1:37121/ using document root <none> and password file <none>
I20260925 05:14:25.345314 23949 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:25.345388 23949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:25.345552 23949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:25.346654 23949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "cc9296b236d24ce6bc769a09d7460cd9"
format_stamp: "Formatted at 2026-09-25 05:14:25 on dist-test-slave-1lvv"
server_key: "69cb52ece2e4bb3b48f8e92233617105"
server_key_iv: "e1ca465f9637c447bd62d982126f517a"
server_key_version: "encryptionkey@0"
I20260925 05:14:25.347033 23949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "cc9296b236d24ce6bc769a09d7460cd9"
format_stamp: "Formatted at 2026-09-25 05:14:25 on dist-test-slave-1lvv"
server_key: "69cb52ece2e4bb3b48f8e92233617105"
server_key_iv: "e1ca465f9637c447bd62d982126f517a"
server_key_version: "encryptionkey@0"
I20260925 05:14:25.348565 23949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:14:25.349329 23964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:25.349507 23949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:25.349617 23949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "cc9296b236d24ce6bc769a09d7460cd9"
format_stamp: "Formatted at 2026-09-25 05:14:25 on dist-test-slave-1lvv"
server_key: "69cb52ece2e4bb3b48f8e92233617105"
server_key_iv: "e1ca465f9637c447bd62d982126f517a"
server_key_version: "encryptionkey@0"
I20260925 05:14:25.349694 23949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:25.367270 23949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:25.369467 23949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:25.369643 23949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:25.373878 23949 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41421
I20260925 05:14:25.373917 24016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41421 every 8 connection(s)
I20260925 05:14:25.374298 23949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:25.375326 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:25.377815 24017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9: Bootstrap starting.
I20260925 05:14:25.378454 24017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:25.378751 24017 log.cc:824] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:25.379554 24017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9: No bootstrap required, opened a new log
I20260925 05:14:25.381194 24017 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9296b236d24ce6bc769a09d7460cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260925 05:14:25.381340 24017 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:25.381364 24017 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9296b236d24ce6bc769a09d7460cd9, State: Initialized, Role: FOLLOWER
I20260925 05:14:25.381554 24017 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9296b236d24ce6bc769a09d7460cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260925 05:14:25.381621 24017 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:25.381644 24017 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:25.381713 24017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:25.382283 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 23949
I20260925 05:14:25.382407 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:25.382505 24017 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9296b236d24ce6bc769a09d7460cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260925 05:14:25.382642 3098 external_mini_cluster.cc:1428] Setting key 43e178c6c8ce911162d2c308194b5b2f
I20260925 05:14:25.382622 24017 leader_election.cc:302] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc9296b236d24ce6bc769a09d7460cd9; no voters:
I20260925 05:14:25.382797 24017 leader_election.cc:288] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:25.382839 24020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:25.383030 24020 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [term 1 LEADER]: Becoming Leader. State: Replica: cc9296b236d24ce6bc769a09d7460cd9, State: Running, Role: LEADER
I20260925 05:14:25.383071 24017 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:25.383185 24020 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9296b236d24ce6bc769a09d7460cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260925 05:14:25.383553 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc9296b236d24ce6bc769a09d7460cd9. Latest consensus state: current_term: 1 leader_uuid: "cc9296b236d24ce6bc769a09d7460cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9296b236d24ce6bc769a09d7460cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } } }
I20260925 05:14:25.383811 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:25.383725 24021 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc9296b236d24ce6bc769a09d7460cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9296b236d24ce6bc769a09d7460cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } } }
I20260925 05:14:25.384075 24021 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:25.384274 24031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:25.384795 24031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:25.386477 24031 catalog_manager.cc:1347] Generated new cluster ID: 6fc65fe5c7584860b671ed4f824b003b
I20260925 05:14:25.386529 24031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:14:25 dist-test-slave-1lvv krb5kdc[23936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:14:25.396346 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:25.383729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:40143 (local address 127.0.0.1:41421)
0925 05:14:25.384225 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:25.384236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:25.384272 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:14:25.384395 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:25.384402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:25.384463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:25.384613 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:25.385907 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:25.387330 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:25.388454 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:25.389135 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:25.389254 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:25.392279 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:25.392283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:25.392284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:25.392297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:25.394361 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:25.394871 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:25.394874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:25.394875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:25.394987 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:25.395235 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:25.395236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:25.395236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:25.395563 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:25.395724 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:25.395891 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 05:14:25.396101 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":151,"threads_started":1}
I20260925 05:14:25.397388 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:25.406602 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:25.399012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:53631 (local address 127.0.0.1:41421)
0925 05:14:25.399042 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:25.399044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:25.399053 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:14:25.399130 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:25.399131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:25.399141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:25.399193 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:25.399906 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:25.400694 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:25.401736 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:25.402048 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:25.402108 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:25.403601 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:25.403603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:25.403603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:25.403610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:25.405194 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:25.405680 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:25.405681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:25.405681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:25.405738 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:25.405977 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:25.405977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:25.405977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:25.406095 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:25.406187 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:25.406350 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:14:25.406411 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 05:14:25.422984 24031 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:25.423789 24031 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:25.429013 24031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc9296b236d24ce6bc769a09d7460cd9: Generated new TSK 0
I20260925 05:14:25.429248 24031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:25.440445 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 23949
W20260925 05:14:25.445518 23934 connection.cc:535] client connection to 127.0.0.1:41421 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41421): Connection reset by peer (error 104)
2026-09-25T05:14:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:25 dist-test-slave-1lvv krb5kdc[24048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:25 dist-test-slave-1lvv krb5kdc[24048](info): set up 2 sockets
Sep 25 05:14:25 dist-test-slave-1lvv krb5kdc[24048](info): commencing operation
krb5kdc: starting...
W20260925 05:14:26.476294 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:26 dist-test-slave-1lvv krb5kdc[24048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:26.608074 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:26.692307 24061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:26.692612 24061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:26.692665 24061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:26.693980 24061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:26.694061 24061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:26.694087 24061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:26.694110 24061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:26.694136 24061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:26.695868 24061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:26.696166 24061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:26.696422 24061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:26.701500 24067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:26.701538 24066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:26.701581 24069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:26.701707 24061 server_base.cc:1034] running on GCE node
I20260925 05:14:26.701956 24061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:26.702280 24061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:26.703459 24061 hybrid_clock.cc:648] HybridClock initialized: now 1790313266703442 us; error 40 us; skew 500 ppm
Sep 25 05:14:26 dist-test-slave-1lvv krb5kdc[24048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:26.706055 24061 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:26.706557 24061 webserver.cc:460] Webserver started at http://127.0.0.1:36889/ using document root <none> and password file <none>
I20260925 05:14:26.706799 24061 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:26.706869 24061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:26.707031 24061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:26.708328 24061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "9c787e67c74c4716a398cb4d3e3ea149"
format_stamp: "Formatted at 2026-09-25 05:14:26 on dist-test-slave-1lvv"
server_key: "d7fd631524b7b7d233b946a0fcb3ad4f"
server_key_iv: "05803f9a02a21361bc0594a67ad99372"
server_key_version: "encryptionkey@0"
I20260925 05:14:26.708760 24061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "9c787e67c74c4716a398cb4d3e3ea149"
format_stamp: "Formatted at 2026-09-25 05:14:26 on dist-test-slave-1lvv"
server_key: "d7fd631524b7b7d233b946a0fcb3ad4f"
server_key_iv: "05803f9a02a21361bc0594a67ad99372"
server_key_version: "encryptionkey@0"
I20260925 05:14:26.710284 24061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:26.711122 24076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:26.711329 24061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:26.711441 24061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "9c787e67c74c4716a398cb4d3e3ea149"
format_stamp: "Formatted at 2026-09-25 05:14:26 on dist-test-slave-1lvv"
server_key: "d7fd631524b7b7d233b946a0fcb3ad4f"
server_key_iv: "05803f9a02a21361bc0594a67ad99372"
server_key_version: "encryptionkey@0"
I20260925 05:14:26.711524 24061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:26.719643 24061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:26.722015 24061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:26.722219 24061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:26.726739 24061 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32969
I20260925 05:14:26.726788 24128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32969 every 8 connection(s)
I20260925 05:14:26.727224 24061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:26.728463 24129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:26.730894 24129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149: Bootstrap starting.
I20260925 05:14:26.731518 24129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:26.731858 24129 log.cc:824] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:26.732622 24129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149: No bootstrap required, opened a new log
I20260925 05:14:26.733804 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24061
I20260925 05:14:26.733973 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:26.734175 3098 external_mini_cluster.cc:1428] Setting key fdd7493f0e9d9df819936c8ad6998765
I20260925 05:14:26.734483 24129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c787e67c74c4716a398cb4d3e3ea149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260925 05:14:26.734788 24129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:26.734863 24129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c787e67c74c4716a398cb4d3e3ea149, State: Initialized, Role: FOLLOWER
I20260925 05:14:26.735016 24129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c787e67c74c4716a398cb4d3e3ea149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260925 05:14:26.735088 24129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:26.735111 24129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:26.735183 24129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:26.735975 24129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c787e67c74c4716a398cb4d3e3ea149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260925 05:14:26.736074 24129 leader_election.cc:302] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c787e67c74c4716a398cb4d3e3ea149; no voters:
I20260925 05:14:26.736263 24129 leader_election.cc:288] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:26.736292 24134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:26.736402 24134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [term 1 LEADER]: Becoming Leader. State: Replica: 9c787e67c74c4716a398cb4d3e3ea149, State: Running, Role: LEADER
I20260925 05:14:26.736541 24129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:26.736573 24134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c787e67c74c4716a398cb4d3e3ea149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260925 05:14:26.736917 24134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c787e67c74c4716a398cb4d3e3ea149. Latest consensus state: current_term: 1 leader_uuid: "9c787e67c74c4716a398cb4d3e3ea149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c787e67c74c4716a398cb4d3e3ea149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260925 05:14:26.737177 24134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:26.737071 24135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c787e67c74c4716a398cb4d3e3ea149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c787e67c74c4716a398cb4d3e3ea149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260925 05:14:26.737468 24135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:26.737886 24142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:26.738437 24142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:26.740764 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:26.734756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:38885 (local address 127.0.0.1:32969)
0925 05:14:26.735059 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:26.735067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:26.735116 (+ 49us) server_negotiation.cc:407] Connection header received
0925 05:14:26.735274 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:26.735282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:26.735343 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:26.735459 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:26.735828 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:26.737123 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:26.738478 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:26.739055 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:26.739170 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:26.739996 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:26.740000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:26.740000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:26.740198 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:26.740228 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:26.740239 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:26.740340 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:26.740489 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 05:14:26.740525 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":116,"threads_started":1}
I20260925 05:14:26.740767 24142 catalog_manager.cc:1347] Generated new cluster ID: fad4132c26014a66ac70b21c8513d4ca
I20260925 05:14:26.740856 24142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:26.741806 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:26.746958 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:26.743394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:35253 (local address 127.0.0.1:32969)
0925 05:14:26.743433 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:26.743435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:26.743444 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:14:26.743514 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:26.743515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:26.743527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:26.743577 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:26.743720 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:26.744503 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:26.745520 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:26.745838 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:26.745912 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:26.746637 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:26.746639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:26.746639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:26.746646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:26.746665 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:26.746671 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:26.746734 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:26.746854 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 05:14:26.746870 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:14:26.752490 24142 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:26.753244 24142 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:26.764256 24142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c787e67c74c4716a398cb4d3e3ea149: Generated new TSK 0
I20260925 05:14:26.764523 24142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:26.768784 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24061
W20260925 05:14:26.773620 24046 connection.cc:535] client connection to 127.0.0.1:32969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32969): Connection reset by peer (error 104)
2026-09-25T05:14:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:26 dist-test-slave-1lvv krb5kdc[24160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:26 dist-test-slave-1lvv krb5kdc[24160](info): set up 2 sockets
Sep 25 05:14:26 dist-test-slave-1lvv krb5kdc[24160](info): commencing operation
krb5kdc: starting...
W20260925 05:14:27.820192 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:27 dist-test-slave-1lvv krb5kdc[24160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:27.950340 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:28.034655 24173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:28.034940 24173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:28.034986 24173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:28.036235 24173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:28.036309 24173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:28.036334 24173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:28.036365 24173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:28.036394 24173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:28.037954 24173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:28.038250 24173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:28.038516 24173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:28.043744 24178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:28.043744 24181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:28.044035 24173 server_base.cc:1034] running on GCE node
W20260925 05:14:28.043744 24179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:28.044427 24173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:28.044759 24173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:28.045931 24173 hybrid_clock.cc:648] HybridClock initialized: now 1790313268045920 us; error 44 us; skew 500 ppm
Sep 25 05:14:28 dist-test-slave-1lvv krb5kdc[24160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:28.052348 24173 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:28.052851 24173 webserver.cc:460] Webserver started at http://127.0.0.1:34863/ using document root <none> and password file <none>
I20260925 05:14:28.053100 24173 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:28.053187 24173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:28.053349 24173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:28.054529 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "f0fdda46ea2141b3a42bff42d7f689ee"
format_stamp: "Formatted at 2026-09-25 05:14:28 on dist-test-slave-1lvv"
server_key: "9c8a1c3b860cf2a0a84b8f3b56c87e71"
server_key_iv: "bd41860a9787066e194bf3827c94eb13"
server_key_version: "encryptionkey@0"
I20260925 05:14:28.054932 24173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "f0fdda46ea2141b3a42bff42d7f689ee"
format_stamp: "Formatted at 2026-09-25 05:14:28 on dist-test-slave-1lvv"
server_key: "9c8a1c3b860cf2a0a84b8f3b56c87e71"
server_key_iv: "bd41860a9787066e194bf3827c94eb13"
server_key_version: "encryptionkey@0"
I20260925 05:14:28.056592 24173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:28.057484 24188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:28.057709 24173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:28.057830 24173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "f0fdda46ea2141b3a42bff42d7f689ee"
format_stamp: "Formatted at 2026-09-25 05:14:28 on dist-test-slave-1lvv"
server_key: "9c8a1c3b860cf2a0a84b8f3b56c87e71"
server_key_iv: "bd41860a9787066e194bf3827c94eb13"
server_key_version: "encryptionkey@0"
I20260925 05:14:28.057912 24173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:28.061554 24173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:28.063509 24173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:28.063697 24173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:28.067682 24173 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40577
I20260925 05:14:28.067718 24240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40577 every 8 connection(s)
I20260925 05:14:28.068135 24173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:28.069126 24241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:28.071426 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee: Bootstrap starting.
I20260925 05:14:28.072156 24241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:28.072449 24241 log.cc:824] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:28.073169 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee: No bootstrap required, opened a new log
I20260925 05:14:28.074759 24241 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260925 05:14:28.074905 24241 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:28.074952 24241 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fdda46ea2141b3a42bff42d7f689ee, State: Initialized, Role: FOLLOWER
I20260925 05:14:28.075145 24241 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260925 05:14:28.075228 24241 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:28.075278 24241 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:28.075332 24241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:28.075982 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24173
I20260925 05:14:28.075978 24241 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260925 05:14:28.076114 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:28.076108 24241 leader_election.cc:302] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0fdda46ea2141b3a42bff42d7f689ee; no voters:
I20260925 05:14:28.076318 3098 external_mini_cluster.cc:1428] Setting key b6a03611ac26d88a8261a5117ce2545b
I20260925 05:14:28.076323 24241 leader_election.cc:288] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:28.076350 24244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:28.076541 24244 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [term 1 LEADER]: Becoming Leader. State: Replica: f0fdda46ea2141b3a42bff42d7f689ee, State: Running, Role: LEADER
I20260925 05:14:28.076597 24241 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:28.076707 24244 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260925 05:14:28.077381 24245 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } } }
I20260925 05:14:28.077474 24245 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:28.077651 24246 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0fdda46ea2141b3a42bff42d7f689ee. Latest consensus state: current_term: 1 leader_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fdda46ea2141b3a42bff42d7f689ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } } }
I20260925 05:14:28.077711 24246 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:28.077888 24255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:28.078521 24255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:28.078997 24248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:28.076973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:49861 (local address 127.0.0.1:40577)
0925 05:14:28.077170 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:28.077182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:28.077218 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:14:28.077345 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:28.077352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:28.077385 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:28.077826 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:28.078344 (+ 518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:28.078347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:28.078348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:28.078351 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:28.078390 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:28.078405 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:28.078686 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:28.078724 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 05:14:28.078822 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260925 05:14:28.080046 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:28.080307 24255 catalog_manager.cc:1347] Generated new cluster ID: 0cc3ff6ca40c4f21bf56f4072631904a
I20260925 05:14:28.080354 24255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:28.088768 24248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:28.087473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:60179 (local address 127.0.0.1:40577)
0925 05:14:28.087547 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:28.087549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:28.087652 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:14:28.087729 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:28.087730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:28.087736 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:28.087806 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:28.088499 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:28.088500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:28.088501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:28.088504 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:28.088516 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:28.088520 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:28.088573 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:28.088660 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 05:14:28.088674 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 05:14:28.094094 24255 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:28.094296 24255 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:28.117304 24255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0fdda46ea2141b3a42bff42d7f689ee: Generated new TSK 0
I20260925 05:14:28.117543 24255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:28.125474 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24173
2026-09-25T05:14:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:28 dist-test-slave-1lvv krb5kdc[24272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:28 dist-test-slave-1lvv krb5kdc[24272](info): set up 2 sockets
Sep 25 05:14:28 dist-test-slave-1lvv krb5kdc[24272](info): commencing operation
krb5kdc: starting...
W20260925 05:14:29.152185 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:29 dist-test-slave-1lvv krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:29.287563 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:29.374015 24285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:29.374248 24285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:29.374274 24285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:29.375494 24285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:29.375546 24285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:29.375561 24285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:29.375574 24285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:29.375586 24285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:29.377252 24285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:29.377477 24285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:29.377699 24285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:29.382889 24290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:29.382885 24293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:29.382882 24291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:29.383085 24285 server_base.cc:1034] running on GCE node
I20260925 05:14:29.383443 24285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:29.383837 24285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:29.384990 24285 hybrid_clock.cc:648] HybridClock initialized: now 1790313269384969 us; error 42 us; skew 500 ppm
Sep 25 05:14:29 dist-test-slave-1lvv krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:29.387825 24285 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:29.388309 24285 webserver.cc:460] Webserver started at http://127.0.0.1:38379/ using document root <none> and password file <none>
I20260925 05:14:29.388592 24285 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:29.388671 24285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:29.388828 24285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:29.389962 24285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "c247a18e36b947deae6f9285d6078219"
format_stamp: "Formatted at 2026-09-25 05:14:29 on dist-test-slave-1lvv"
server_key: "0d88d402d3f4dc7b0fff933c3671d2ef"
server_key_iv: "4ce219bea8cc0baf1b5222d4796c82ec"
server_key_version: "encryptionkey@0"
I20260925 05:14:29.390348 24285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "c247a18e36b947deae6f9285d6078219"
format_stamp: "Formatted at 2026-09-25 05:14:29 on dist-test-slave-1lvv"
server_key: "0d88d402d3f4dc7b0fff933c3671d2ef"
server_key_iv: "4ce219bea8cc0baf1b5222d4796c82ec"
server_key_version: "encryptionkey@0"
I20260925 05:14:29.391866 24285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 05:14:29.392722 24300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:29.392936 24285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:29.393054 24285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "c247a18e36b947deae6f9285d6078219"
format_stamp: "Formatted at 2026-09-25 05:14:29 on dist-test-slave-1lvv"
server_key: "0d88d402d3f4dc7b0fff933c3671d2ef"
server_key_iv: "4ce219bea8cc0baf1b5222d4796c82ec"
server_key_version: "encryptionkey@0"
I20260925 05:14:29.393146 24285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:29.411875 24285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:29.414125 24285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:29.414320 24285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:29.418797 24285 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34419
I20260925 05:14:29.418860 24352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34419 every 8 connection(s)
I20260925 05:14:29.419260 24285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:29.420405 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:29.422860 24353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219: Bootstrap starting.
I20260925 05:14:29.423593 24353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:29.423743 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24285
I20260925 05:14:29.423909 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:29.423998 24353 log.cc:824] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:29.424127 3098 external_mini_cluster.cc:1428] Setting key 27a2fe28f9def65125d5b9161c5bf8c5
I20260925 05:14:29.424939 24353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219: No bootstrap required, opened a new log
I20260925 05:14:29.427232 24353 raft_consensus.cc:348] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247a18e36b947deae6f9285d6078219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34419 } }
I20260925 05:14:29.427419 24353 raft_consensus.cc:374] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:29.427474 24353 raft_consensus.cc:729] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c247a18e36b947deae6f9285d6078219, State: Initialized, Role: FOLLOWER
I20260925 05:14:29.427624 24353 consensus_queue.cc:260] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247a18e36b947deae6f9285d6078219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34419 } }
I20260925 05:14:29.427706 24353 raft_consensus.cc:388] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:29.427731 24353 raft_consensus.cc:482] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:29.427845 24353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:29.428475 24353 raft_consensus.cc:504] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247a18e36b947deae6f9285d6078219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34419 } }
I20260925 05:14:29.428606 24353 leader_election.cc:302] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c247a18e36b947deae6f9285d6078219; no voters:
I20260925 05:14:29.428805 24353 leader_election.cc:288] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:29.428861 24358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:29.429025 24358 raft_consensus.cc:686] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [term 1 LEADER]: Becoming Leader. State: Replica: c247a18e36b947deae6f9285d6078219, State: Running, Role: LEADER
I20260925 05:14:29.429105 24353 sys_catalog.cc:564] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:29.429175 24358 consensus_queue.cc:237] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247a18e36b947deae6f9285d6078219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34419 } }
W20260925 05:14:29.430930 24372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:29.431062 24372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:29.431059 24360 sys_catalog.cc:455] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c247a18e36b947deae6f9285d6078219. Latest consensus state: current_term: 1 leader_uuid: "c247a18e36b947deae6f9285d6078219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247a18e36b947deae6f9285d6078219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34419 } } }
I20260925 05:14:29.431224 24359 sys_catalog.cc:455] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c247a18e36b947deae6f9285d6078219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247a18e36b947deae6f9285d6078219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34419 } } }
I20260925 05:14:29.431316 24359 sys_catalog.cc:458] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:29.431510 24360 sys_catalog.cc:458] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219 [sys.catalog]: This master's current role is: LEADER
Sep 25 05:14:29 dist-test-slave-1lvv krb5kdc[24272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:14:29.432098 24374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:29.432333 24374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:29.434031 24374 catalog_manager.cc:1347] Generated new cluster ID: d8da300bd38a47d2a806e8b785aab275
I20260925 05:14:29.434082 24374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:29.436534 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:29.424673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:39491 (local address 127.0.0.1:34419)
0925 05:14:29.424874 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:29.424885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:29.424935 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:14:29.425479 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:29.425487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:29.425547 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:29.425714 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:29.426400 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:29.427316 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:29.428315 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:29.428830 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:29.428958 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:29.432268 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:29.432272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:29.432272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:29.432286 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:29.434495 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:29.434939 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:29.434942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:29.434943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:29.435050 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:29.435270 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:29.435270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:29.435271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:29.435561 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:29.435719 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:29.436235 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 05:14:29.436335 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260925 05:14:29.440728 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:29.446211 24374 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:29.446802 24374 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:29.450341 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:29.442406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:60119 (local address 127.0.0.1:34419)
0925 05:14:29.442458 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:29.442460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:29.442657 (+ 197us) server_negotiation.cc:407] Connection header received
0925 05:14:29.442723 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:29.442724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:29.442735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:29.442789 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:29.443498 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:29.444308 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:29.445404 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:29.445721 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:29.445794 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:29.447375 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:29.447376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:29.447376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:29.447383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:29.449037 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:29.449497 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:29.449498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:29.449498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:29.449556 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:29.449793 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:29.449793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:29.449794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:29.449914 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:29.450014 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:29.450184 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 05:14:29.450248 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:14:29.453684 24374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c247a18e36b947deae6f9285d6078219: Generated new TSK 0
I20260925 05:14:29.453879 24374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:29.461687 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24285
W20260925 05:14:29.466511 24270 connection.cc:535] client connection to 127.0.0.1:34419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34419): Connection reset by peer (error 104)
2026-09-25T05:14:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:29 dist-test-slave-1lvv krb5kdc[24384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:29 dist-test-slave-1lvv krb5kdc[24384](info): set up 2 sockets
Sep 25 05:14:29 dist-test-slave-1lvv krb5kdc[24384](info): commencing operation
krb5kdc: starting...
W20260925 05:14:30.508309 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:30 dist-test-slave-1lvv krb5kdc[24384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:30.643205 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:30.726261 24397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:30.726553 24397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:30.726599 24397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:30.727870 24397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:30.727945 24397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:30.727977 24397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:30.728010 24397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:30.728039 24397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:30.729590 24397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:30.729874 24397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:30.730120 24397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:30.735321 24402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:30.735330 24405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:30.735330 24403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:30.735404 24397 server_base.cc:1034] running on GCE node
I20260925 05:14:30.735908 24397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:30.736240 24397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:30.737389 24397 hybrid_clock.cc:648] HybridClock initialized: now 1790313270737370 us; error 38 us; skew 500 ppm
Sep 25 05:14:30 dist-test-slave-1lvv krb5kdc[24384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:30.739989 24397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:30.740427 24397 webserver.cc:460] Webserver started at http://127.0.0.1:33131/ using document root <none> and password file <none>
I20260925 05:14:30.740650 24397 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:30.740696 24397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:30.740792 24397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:30.741896 24397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "a5c77f71646b4642a219f288271dbf8e"
format_stamp: "Formatted at 2026-09-25 05:14:30 on dist-test-slave-1lvv"
server_key: "a943a42be609b437dba996d76079d18a"
server_key_iv: "417c4392e713ef5d668b7e719743d85a"
server_key_version: "encryptionkey@0"
I20260925 05:14:30.742239 24397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "a5c77f71646b4642a219f288271dbf8e"
format_stamp: "Formatted at 2026-09-25 05:14:30 on dist-test-slave-1lvv"
server_key: "a943a42be609b437dba996d76079d18a"
server_key_iv: "417c4392e713ef5d668b7e719743d85a"
server_key_version: "encryptionkey@0"
I20260925 05:14:30.743646 24397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 05:14:30.744541 24412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:30.744738 24397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:30.744836 24397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "a5c77f71646b4642a219f288271dbf8e"
format_stamp: "Formatted at 2026-09-25 05:14:30 on dist-test-slave-1lvv"
server_key: "a943a42be609b437dba996d76079d18a"
server_key_iv: "417c4392e713ef5d668b7e719743d85a"
server_key_version: "encryptionkey@0"
I20260925 05:14:30.744918 24397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:30.763729 24397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:30.766287 24397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:30.766440 24397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:30.770679 24397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41467
I20260925 05:14:30.770714 24464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41467 every 8 connection(s)
I20260925 05:14:30.771097 24397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:30.772091 24465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:30.774457 24465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e: Bootstrap starting.
I20260925 05:14:30.775090 24465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:30.775394 24465 log.cc:824] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:30.776270 24465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e: No bootstrap required, opened a new log
I20260925 05:14:30.777921 24465 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c77f71646b4642a219f288271dbf8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260925 05:14:30.778069 24465 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:30.778095 24465 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c77f71646b4642a219f288271dbf8e, State: Initialized, Role: FOLLOWER
I20260925 05:14:30.778254 24465 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c77f71646b4642a219f288271dbf8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260925 05:14:30.778337 24465 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:30.778381 24465 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:30.778434 24465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:30.779062 24465 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c77f71646b4642a219f288271dbf8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260925 05:14:30.779156 24465 leader_election.cc:302] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5c77f71646b4642a219f288271dbf8e; no voters:
I20260925 05:14:30.779208 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24397
I20260925 05:14:30.779349 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:30.779353 24465 leader_election.cc:288] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:30.779405 24468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:30.779590 3098 external_mini_cluster.cc:1428] Setting key 83698e01cc239e1df183bcfd4a53fba0
I20260925 05:14:30.779603 24468 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [term 1 LEADER]: Becoming Leader. State: Replica: a5c77f71646b4642a219f288271dbf8e, State: Running, Role: LEADER
I20260925 05:14:30.779644 24465 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:30.779696 24468 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c77f71646b4642a219f288271dbf8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260925 05:14:30.780049 24468 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5c77f71646b4642a219f288271dbf8e. Latest consensus state: current_term: 1 leader_uuid: "a5c77f71646b4642a219f288271dbf8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c77f71646b4642a219f288271dbf8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } } }
I20260925 05:14:30.780145 24468 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:30.780303 24469 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5c77f71646b4642a219f288271dbf8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c77f71646b4642a219f288271dbf8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } } }
I20260925 05:14:30.780418 24469 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:30.780469 24475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:30.781045 24475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:30.783098 24475 catalog_manager.cc:1347] Generated new cluster ID: 11c7c4b06498480680c715b7591ee1eb
I20260925 05:14:30.783152 24475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:30.786324 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:30.780310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:53091 (local address 127.0.0.1:41467)
0925 05:14:30.780512 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:30.780522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:30.780807 (+ 285us) server_negotiation.cc:407] Connection header received
0925 05:14:30.781062 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:30.781083 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:30.781152 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:30.781282 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:30.781645 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:30.782825 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:30.783995 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:30.784618 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:30.784729 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:30.785603 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:30.785607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:30.785607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:30.785816 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:30.785854 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:30.785867 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:30.785949 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:30.786077 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:14:30.786106 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260925 05:14:30.787401 24475 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:30.787416 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:30.787979 24475 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:30.792834 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:30.789248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:53353 (local address 127.0.0.1:41467)
0925 05:14:30.789288 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:30.789290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:30.789297 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:14:30.789380 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:30.789380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:30.789390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:30.789423 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:30.789600 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:30.790335 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:30.791363 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:30.791684 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:30.791722 (+ 38us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:30.792560 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:30.792561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:30.792561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:30.792566 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:30.792592 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:30.792596 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:30.792627 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:30.792742 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 05:14:30.792756 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:14:30.794384 24475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5c77f71646b4642a219f288271dbf8e: Generated new TSK 0
I20260925 05:14:30.794574 24475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:30.800727 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24397
W20260925 05:14:30.805387 24382 connection.cc:535] client connection to 127.0.0.1:41467 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41467): Connection reset by peer (error 104)
2026-09-25T05:14:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:30 dist-test-slave-1lvv krb5kdc[24496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:30 dist-test-slave-1lvv krb5kdc[24496](info): set up 2 sockets
Sep 25 05:14:30 dist-test-slave-1lvv krb5kdc[24496](info): commencing operation
krb5kdc: starting...
W20260925 05:14:31.872404 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:31 dist-test-slave-1lvv krb5kdc[24496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:14:32.007966 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:32.093803 24509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:32.094070 24509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:32.094113 24509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:32.095346 24509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:32.095427 24509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:32.095451 24509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:32.095474 24509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:32.095510 24509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:32.097289 24509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:32.097606 24509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:32.097860 24509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:32.103089 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:32.103097 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:32.103207 24509 server_base.cc:1034] running on GCE node
W20260925 05:14:32.103097 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:32.103600 24509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:32.103969 24509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:32.105119 24509 hybrid_clock.cc:648] HybridClock initialized: now 1790313272105092 us; error 58 us; skew 500 ppm
Sep 25 05:14:32 dist-test-slave-1lvv krb5kdc[24496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:32.107955 24509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:14:32.108511 24509 webserver.cc:460] Webserver started at http://127.0.0.1:35111/ using document root <none> and password file <none>
I20260925 05:14:32.108798 24509 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:32.108848 24509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:32.109030 24509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:32.110453 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "282844f916984de9b51909f7907019ee"
format_stamp: "Formatted at 2026-09-25 05:14:32 on dist-test-slave-1lvv"
server_key: "13b2c30ef45427921b09a116c9f39de9"
server_key_iv: "3090759b7b08a9411ccb80157140bdb4"
server_key_version: "encryptionkey@0"
I20260925 05:14:32.110958 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "282844f916984de9b51909f7907019ee"
format_stamp: "Formatted at 2026-09-25 05:14:32 on dist-test-slave-1lvv"
server_key: "13b2c30ef45427921b09a116c9f39de9"
server_key_iv: "3090759b7b08a9411ccb80157140bdb4"
server_key_version: "encryptionkey@0"
I20260925 05:14:32.112598 24509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 05:14:32.113428 24524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:32.113626 24509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:32.113760 24509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "282844f916984de9b51909f7907019ee"
format_stamp: "Formatted at 2026-09-25 05:14:32 on dist-test-slave-1lvv"
server_key: "13b2c30ef45427921b09a116c9f39de9"
server_key_iv: "3090759b7b08a9411ccb80157140bdb4"
server_key_version: "encryptionkey@0"
I20260925 05:14:32.113840 24509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:32.117440 24509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:32.119449 24509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:32.119616 24509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:32.124073 24509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46517
I20260925 05:14:32.124127 24576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46517 every 8 connection(s)
I20260925 05:14:32.124503 24509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:32.125627 24577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:32.128203 24577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee: Bootstrap starting.
I20260925 05:14:32.128799 24577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:32.129050 24577 log.cc:824] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:32.129954 24577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee: No bootstrap required, opened a new log
I20260925 05:14:32.131685 24577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282844f916984de9b51909f7907019ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46517 } }
I20260925 05:14:32.131916 24577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:32.131946 24577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282844f916984de9b51909f7907019ee, State: Initialized, Role: FOLLOWER
I20260925 05:14:32.132097 24577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282844f916984de9b51909f7907019ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46517 } }
I20260925 05:14:32.132179 24577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:32.132202 24577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:32.132232 24577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:32.132813 24577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282844f916984de9b51909f7907019ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46517 } }
I20260925 05:14:32.132905 24577 leader_election.cc:302] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 282844f916984de9b51909f7907019ee; no voters:
I20260925 05:14:32.133047 24577 leader_election.cc:288] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:32.133109 24580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:32.133343 24577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:32.133348 24580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [term 1 LEADER]: Becoming Leader. State: Replica: 282844f916984de9b51909f7907019ee, State: Running, Role: LEADER
I20260925 05:14:32.133543 24580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282844f916984de9b51909f7907019ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46517 } }
I20260925 05:14:32.134052 24582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282844f916984de9b51909f7907019ee. Latest consensus state: current_term: 1 leader_uuid: "282844f916984de9b51909f7907019ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282844f916984de9b51909f7907019ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46517 } } }
I20260925 05:14:32.134151 24582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:32.134265 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24509
I20260925 05:14:32.134421 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:32.134632 3098 external_mini_cluster.cc:1428] Setting key 3998e924de7e0db831238b3ce3d9b7c3
I20260925 05:14:32.134052 24581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "282844f916984de9b51909f7907019ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282844f916984de9b51909f7907019ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46517 } } }
I20260925 05:14:32.134917 24581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:32.135155 24584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:32.135823 24584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:32.137552 24584 catalog_manager.cc:1347] Generated new cluster ID: 1d07037d05274792ab6ba8cc282638c3
I20260925 05:14:32.137610 24584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:32.138025 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:32.135432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:37067 (local address 127.0.0.1:46517)
0925 05:14:32.135602 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:32.135613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:32.135649 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:14:32.135855 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:32.135863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:32.136038 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:32.136280 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:32.137532 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:32.137536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:32.137536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:32.137542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:32.137594 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:32.137610 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:32.137699 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:32.137784 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 05:14:32.137834 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 05:14:32.138949 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:32.145704 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:32.144236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:38499 (local address 127.0.0.1:46517)
0925 05:14:32.144293 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:32.144295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:32.144571 (+ 276us) server_negotiation.cc:407] Connection header received
0925 05:14:32.144664 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:32.144665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:32.144672 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:32.144747 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:32.145458 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:32.145459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:32.145460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:32.145464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:32.145481 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:32.145485 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:32.145540 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:32.145592 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 05:14:32.145609 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:14:32.149813 24584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:32.150043 24584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:32.160585 24584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 282844f916984de9b51909f7907019ee: Generated new TSK 0
I20260925 05:14:32.160849 24584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:32.164698 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24509
2026-09-25T05:14:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:32 dist-test-slave-1lvv krb5kdc[24608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:32 dist-test-slave-1lvv krb5kdc[24608](info): set up 2 sockets
Sep 25 05:14:32 dist-test-slave-1lvv krb5kdc[24608](info): commencing operation
krb5kdc: starting...
W20260925 05:14:33.212242 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:33 dist-test-slave-1lvv krb5kdc[24608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:33.347703 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:35661
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:35661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:33.436498 24621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:33.436777 24621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:33.436816 24621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:33.438017 24621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:33.438092 24621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:33.438117 24621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:33.438138 24621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:33.438164 24621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:33.439673 24621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:35661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:33.439973 24621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:33.440218 24621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:33.445333 24626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:33.445394 24621 server_base.cc:1034] running on GCE node
W20260925 05:14:33.445339 24627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:33.445350 24629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:33.445838 24621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:33.446147 24621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:33.447296 24621 hybrid_clock.cc:648] HybridClock initialized: now 1790313273447280 us; error 36 us; skew 500 ppm
Sep 25 05:14:33 dist-test-slave-1lvv krb5kdc[24608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:33.456373 24621 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:33.456890 24621 webserver.cc:460] Webserver started at http://127.3.6.190:40423/ using document root <none> and password file <none>
I20260925 05:14:33.457101 24621 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:33.457145 24621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:33.457247 24621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:33.458482 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "d04c7083f76a47f9a504b029aaba0eef"
format_stamp: "Formatted at 2026-09-25 05:14:33 on dist-test-slave-1lvv"
server_key: "2cbb867b54ed0c963024429e3ebdb414"
server_key_iv: "57e01ea6dc5058c3e52cabaf830fbef0"
server_key_version: "encryptionkey@0"
I20260925 05:14:33.458823 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "d04c7083f76a47f9a504b029aaba0eef"
format_stamp: "Formatted at 2026-09-25 05:14:33 on dist-test-slave-1lvv"
server_key: "2cbb867b54ed0c963024429e3ebdb414"
server_key_iv: "57e01ea6dc5058c3e52cabaf830fbef0"
server_key_version: "encryptionkey@0"
I20260925 05:14:33.460304 24621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:33.461061 24636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:33.461275 24621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:33.461393 24621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "d04c7083f76a47f9a504b029aaba0eef"
format_stamp: "Formatted at 2026-09-25 05:14:33 on dist-test-slave-1lvv"
server_key: "2cbb867b54ed0c963024429e3ebdb414"
server_key_iv: "57e01ea6dc5058c3e52cabaf830fbef0"
server_key_version: "encryptionkey@0"
I20260925 05:14:33.461486 24621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:33.469419 24621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:33.471901 24621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:33.472082 24621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:33.476467 24621 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:35661
I20260925 05:14:33.476527 24688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:35661 every 8 connection(s)
I20260925 05:14:33.476918 24621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:33.477809 24689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:33.480216 24689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef: Bootstrap starting.
I20260925 05:14:33.480811 24689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:33.481094 24689 log.cc:824] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:33.481824 24689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef: No bootstrap required, opened a new log
I20260925 05:14:33.483378 24689 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04c7083f76a47f9a504b029aaba0eef" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35661 } }
I20260925 05:14:33.483541 24689 raft_consensus.cc:374] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:33.483564 24689 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04c7083f76a47f9a504b029aaba0eef, State: Initialized, Role: FOLLOWER
I20260925 05:14:33.483732 24689 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04c7083f76a47f9a504b029aaba0eef" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35661 } }
I20260925 05:14:33.483868 24689 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:33.483845 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24621
I20260925 05:14:33.483894 24689 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:33.483955 24689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:33.484009 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:33.484201 3098 external_mini_cluster.cc:1428] Setting key 0691ac517ec726bc1a0e68b414979e3e
I20260925 05:14:33.485137 24689 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04c7083f76a47f9a504b029aaba0eef" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35661 } }
I20260925 05:14:33.485247 24689 leader_election.cc:302] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04c7083f76a47f9a504b029aaba0eef; no voters:
I20260925 05:14:33.485407 24689 leader_election.cc:288] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:33.485461 24694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:33.485651 24694 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [term 1 LEADER]: Becoming Leader. State: Replica: d04c7083f76a47f9a504b029aaba0eef, State: Running, Role: LEADER
I20260925 05:14:33.485740 24689 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:33.485827 24694 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04c7083f76a47f9a504b029aaba0eef" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35661 } }
I20260925 05:14:33.486243 24694 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04c7083f76a47f9a504b029aaba0eef. Latest consensus state: current_term: 1 leader_uuid: "d04c7083f76a47f9a504b029aaba0eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04c7083f76a47f9a504b029aaba0eef" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35661 } } }
I20260925 05:14:33.486346 24694 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:33.486251 24695 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d04c7083f76a47f9a504b029aaba0eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04c7083f76a47f9a504b029aaba0eef" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 35661 } } }
I20260925 05:14:33.486620 24695 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef [sys.catalog]: This master's current role is: LEADER
W20260925 05:14:33.488137 24709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:33.488210 24709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:33.488297 24702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:33.488464 24702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:33.489909 24702 catalog_manager.cc:1347] Generated new cluster ID: d2242ba011f4458b8e02b8f924252a56
I20260925 05:14:33.489964 24702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:14:33 dist-test-slave-1lvv krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:33.496987 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:33.484764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60691 (local address 127.3.6.190:35661)
0925 05:14:33.484989 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:33.485001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:33.485039 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:14:33.485159 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:33.485167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:33.485236 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:33.485388 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:33.486343 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:33.487906 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:33.489020 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:33.489615 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:33.489721 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:33.493082 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:33.493088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:33.493088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:33.493102 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:33.495149 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:33.495644 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:33.495648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:33.495648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:33.495807 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:33.496057 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:33.496057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:33.496058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:33.496373 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:33.496533 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:33.496679 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 05:14:33.496778 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":130,"threads_started":1}
I20260925 05:14:33.498103 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:33.502128 24702 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:33.502662 24702 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:33.505704 24702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d04c7083f76a47f9a504b029aaba0eef: Generated new TSK 0
I20260925 05:14:33.505891 24702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:33.512012 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:33.499626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48487 (local address 127.3.6.190:35661)
0925 05:14:33.499659 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:33.499661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:33.501728 (+ 2067us) server_negotiation.cc:407] Connection header received
0925 05:14:33.501991 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:33.501992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:33.502004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:33.502061 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:33.506329 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:33.506812 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:33.507596 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:33.507925 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:33.508012 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:33.509332 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:33.509333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:33.509333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:33.509341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:33.510708 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:33.511119 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:33.511120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:33.511120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:33.511185 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:33.511421 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:33.511422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:33.511423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:33.511544 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:33.511693 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:33.511837 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 05:14:33.511905 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:14:33.513991 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24621
W20260925 05:14:33.518908 24606 connection.cc:535] client connection to 127.3.6.190:35661 recv error: Network error: failed to read from TLS socket (remote: 127.3.6.190:35661): Connection reset by peer (error 104)
2026-09-25T05:14:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:33 dist-test-slave-1lvv krb5kdc[24720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:33 dist-test-slave-1lvv krb5kdc[24720](info): set up 2 sockets
Sep 25 05:14:33 dist-test-slave-1lvv krb5kdc[24720](info): commencing operation
krb5kdc: starting...
W20260925 05:14:34.560336 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:34 dist-test-slave-1lvv krb5kdc[24720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:34.691509 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40115
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:40115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:34.777673 24733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:34.777910 24733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:34.777937 24733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:34.779093 24733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:34.779134 24733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:34.779150 24733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:34.779163 24733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:34.779176 24733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:34.780740 24733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:40115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:40115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:34.780953 24733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:34.781136 24733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:14:34.786907 24733 server_base.cc:1034] running on GCE node
W20260925 05:14:34.786947 24739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:34.787077 24738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:34.786973 24741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:34.787443 24733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:34.787846 24733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:34.789012 24733 hybrid_clock.cc:648] HybridClock initialized: now 1790313274789003 us; error 34 us; skew 500 ppm
Sep 25 05:14:34 dist-test-slave-1lvv krb5kdc[24720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313274, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:34.791745 24733 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:34.792392 24733 webserver.cc:460] Webserver started at http://127.3.6.190:44491/ using document root <none> and password file <none>
I20260925 05:14:34.792644 24733 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:34.792714 24733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:34.792879 24733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:34.794025 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "4648cb94fcce4ef182404c4e6b413ca9"
format_stamp: "Formatted at 2026-09-25 05:14:34 on dist-test-slave-1lvv"
server_key: "a887e6e8e45178dc74c45167aed5faf3"
server_key_iv: "60e27938d0907911d2916cefc36b948e"
server_key_version: "encryptionkey@0"
I20260925 05:14:34.794423 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "4648cb94fcce4ef182404c4e6b413ca9"
format_stamp: "Formatted at 2026-09-25 05:14:34 on dist-test-slave-1lvv"
server_key: "a887e6e8e45178dc74c45167aed5faf3"
server_key_iv: "60e27938d0907911d2916cefc36b948e"
server_key_version: "encryptionkey@0"
I20260925 05:14:34.796006 24733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:34.796859 24748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:34.797062 24733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:34.797173 24733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "4648cb94fcce4ef182404c4e6b413ca9"
format_stamp: "Formatted at 2026-09-25 05:14:34 on dist-test-slave-1lvv"
server_key: "a887e6e8e45178dc74c45167aed5faf3"
server_key_iv: "60e27938d0907911d2916cefc36b948e"
server_key_version: "encryptionkey@0"
I20260925 05:14:34.797260 24733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:34.827569 24733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:34.830137 24733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:34.830329 24733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:34.834829 24733 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:40115
I20260925 05:14:34.834959 24800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:40115 every 8 connection(s)
I20260925 05:14:34.835295 24733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:34.836321 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:34.838145 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24733
I20260925 05:14:34.838356 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:34.838578 3098 external_mini_cluster.cc:1428] Setting key 82adccc2ce7b52f65eee7b4d84ffd0d9
I20260925 05:14:34.839351 24801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9: Bootstrap starting.
I20260925 05:14:34.840269 24801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:34.840662 24801 log.cc:824] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:34.841609 24801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9: No bootstrap required, opened a new log
I20260925 05:14:34.844172 24801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648cb94fcce4ef182404c4e6b413ca9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40115 } }
I20260925 05:14:34.844404 24801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:34.844473 24801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4648cb94fcce4ef182404c4e6b413ca9, State: Initialized, Role: FOLLOWER
I20260925 05:14:34.844628 24801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648cb94fcce4ef182404c4e6b413ca9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40115 } }
I20260925 05:14:34.844741 24801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:34.844794 24801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:34.844853 24801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:34.845809 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:34.839268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60835 (local address 127.3.6.190:40115)
0925 05:14:34.839509 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:34.839519 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:34.839558 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:14:34.840256 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:34.840266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:34.840339 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:34.840527 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:34.840840 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:34.842263 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:34.843158 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:34.843947 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:34.844095 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:34.844943 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:34.844947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:34.844947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:34.845194 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:34.845239 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:34.845255 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:34.845361 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:34.845521 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 05:14:34.845554 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260925 05:14:34.845809 24801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648cb94fcce4ef182404c4e6b413ca9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40115 } }
I20260925 05:14:34.845943 24801 leader_election.cc:302] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4648cb94fcce4ef182404c4e6b413ca9; no voters:
I20260925 05:14:34.846164 24801 leader_election.cc:288] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:34.846199 24806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:34.846403 24806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [term 1 LEADER]: Becoming Leader. State: Replica: 4648cb94fcce4ef182404c4e6b413ca9, State: Running, Role: LEADER
I20260925 05:14:34.846531 24806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648cb94fcce4ef182404c4e6b413ca9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40115 } }
I20260925 05:14:34.846565 24801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:34.847101 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:34.848852 24808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4648cb94fcce4ef182404c4e6b413ca9. Latest consensus state: current_term: 1 leader_uuid: "4648cb94fcce4ef182404c4e6b413ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648cb94fcce4ef182404c4e6b413ca9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40115 } } }
I20260925 05:14:34.848975 24808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:34.849375 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4648cb94fcce4ef182404c4e6b413ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648cb94fcce4ef182404c4e6b413ca9" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 40115 } } }
I20260925 05:14:34.849571 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9 [sys.catalog]: This master's current role is: LEADER
W20260925 05:14:34.849848 24824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:34.849916 24824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:34.849965 24826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:34.850128 24826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:34.851855 24826 catalog_manager.cc:1347] Generated new cluster ID: 88254414355c43e384f824a6bac19519
I20260925 05:14:34.851923 24826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:34.852708 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:34.849007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40515 (local address 127.3.6.190:40115)
0925 05:14:34.849052 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:34.849054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:34.849187 (+ 133us) server_negotiation.cc:407] Connection header received
0925 05:14:34.849311 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:34.849312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:34.849323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:34.849361 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:34.849627 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:34.850300 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:34.851224 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:34.851530 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:34.851661 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:34.852432 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:34.852434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:34.852434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:34.852439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:34.852456 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:34.852461 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:34.852536 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:34.852621 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 05:14:34.852636 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:14:34.858654 24826 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:34.859416 24826 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:34.862361 24826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4648cb94fcce4ef182404c4e6b413ca9: Generated new TSK 0
I20260925 05:14:34.862562 24826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:34.866019 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24733
W20260925 05:14:34.871145 24718 connection.cc:535] client connection to 127.3.6.190:40115 recv error: Network error: failed to read from TLS socket (remote: 127.3.6.190:40115): Connection reset by peer (error 104)
2026-09-25T05:14:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:34 dist-test-slave-1lvv krb5kdc[24832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:34 dist-test-slave-1lvv krb5kdc[24832](info): set up 2 sockets
Sep 25 05:14:34 dist-test-slave-1lvv krb5kdc[24832](info): commencing operation
krb5kdc: starting...
W20260925 05:14:35.916242 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:36 dist-test-slave-1lvv krb5kdc[24832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:36.047616 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44995
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:36.134336 24845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:36.134645 24845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:36.134691 24845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:36.135988 24845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:36.136075 24845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:36.136104 24845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:36.136132 24845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:36.136164 24845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:36.137804 24845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:44995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:36.138139 24845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:36.138415 24845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:14:36.144006 24845 server_base.cc:1034] running on GCE node
W20260925 05:14:36.143941 24850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:36.143993 24851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:36.143941 24853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:36.144553 24845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:36.144848 24845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:36.146028 24845 hybrid_clock.cc:648] HybridClock initialized: now 1790313276146001 us; error 49 us; skew 500 ppm
Sep 25 05:14:36 dist-test-slave-1lvv krb5kdc[24832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313276, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:36.148953 24845 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:36.149513 24845 webserver.cc:460] Webserver started at http://127.3.6.190:40157/ using document root <none> and password file <none>
I20260925 05:14:36.149768 24845 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:36.149845 24845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:36.150019 24845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:36.151216 24845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "eb2e2fbb25d643a3984af0dcd1bed103"
format_stamp: "Formatted at 2026-09-25 05:14:36 on dist-test-slave-1lvv"
server_key: "b08b30bcdf3e4a38deb00ffec5de075d"
server_key_iv: "626131a4765ad73bd327487df67bf645"
server_key_version: "encryptionkey@0"
I20260925 05:14:36.151636 24845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "eb2e2fbb25d643a3984af0dcd1bed103"
format_stamp: "Formatted at 2026-09-25 05:14:36 on dist-test-slave-1lvv"
server_key: "b08b30bcdf3e4a38deb00ffec5de075d"
server_key_iv: "626131a4765ad73bd327487df67bf645"
server_key_version: "encryptionkey@0"
I20260925 05:14:36.153384 24845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:36.154197 24860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:36.154529 24845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:36.154649 24845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "eb2e2fbb25d643a3984af0dcd1bed103"
format_stamp: "Formatted at 2026-09-25 05:14:36 on dist-test-slave-1lvv"
server_key: "b08b30bcdf3e4a38deb00ffec5de075d"
server_key_iv: "626131a4765ad73bd327487df67bf645"
server_key_version: "encryptionkey@0"
I20260925 05:14:36.154731 24845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:36.158244 24845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:36.160504 24845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:36.160672 24845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:36.164803 24845 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:44995
I20260925 05:14:36.164860 24912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:44995 every 8 connection(s)
I20260925 05:14:36.165228 24845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:36.166252 24913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:36.168855 24913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103: Bootstrap starting.
I20260925 05:14:36.169606 24913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:36.169932 24913 log.cc:824] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:36.170857 24913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103: No bootstrap required, opened a new log
I20260925 05:14:36.172583 24913 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44995 } }
I20260925 05:14:36.172755 24913 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:36.172780 24913 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2e2fbb25d643a3984af0dcd1bed103, State: Initialized, Role: FOLLOWER
I20260925 05:14:36.172991 24913 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44995 } }
I20260925 05:14:36.173080 24913 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:36.173138 24913 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:36.173194 24913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:36.173362 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24845
I20260925 05:14:36.173501 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:36.173684 3098 external_mini_cluster.cc:1428] Setting key 9aa11a96f5146012f49a25d4eff42d77
I20260925 05:14:36.174563 24913 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44995 } }
I20260925 05:14:36.174676 24913 leader_election.cc:302] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2e2fbb25d643a3984af0dcd1bed103; no voters:
I20260925 05:14:36.174870 24913 leader_election.cc:288] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:36.174897 24918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:36.175011 24918 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [term 1 LEADER]: Becoming Leader. State: Replica: eb2e2fbb25d643a3984af0dcd1bed103, State: Running, Role: LEADER
I20260925 05:14:36.175169 24913 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:36.175184 24918 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44995 } }
I20260925 05:14:36.176312 24920 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44995 } } }
I20260925 05:14:36.176524 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:36.174252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58929 (local address 127.3.6.190:44995)
0925 05:14:36.174458 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:36.174470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:36.174508 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:14:36.174626 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:36.174632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:36.174686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:36.174860 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:36.175997 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:36.176002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:36.176002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:36.176007 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:36.176059 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:36.176078 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:36.176134 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:36.176283 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 05:14:36.176329 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260925 05:14:36.176568 24919 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2e2fbb25d643a3984af0dcd1bed103. Latest consensus state: current_term: 1 leader_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e2fbb25d643a3984af0dcd1bed103" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 44995 } } }
W20260925 05:14:36.177340 24933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 05:14:36.177397 24919 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [sys.catalog]: This master's current role is: LEADER
W20260925 05:14:36.177417 24933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:36.177608 24935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:36.177104 24920 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:36.177814 24935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:36.177915 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:36.179672 24935 catalog_manager.cc:1347] Generated new cluster ID: 12d4885135d94fe38762e7041de805b6
I20260925 05:14:36.179744 24935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:36.180776 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:36.179469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42315 (local address 127.3.6.190:44995)
0925 05:14:36.179510 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:36.179512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:36.179624 (+ 112us) server_negotiation.cc:407] Connection header received
0925 05:14:36.179717 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:36.179718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:36.179725 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:36.179828 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:36.180541 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:36.180542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:36.180542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:36.180546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:36.180563 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:36.180567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:36.180622 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:36.180702 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:14:36.180716 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:14:36.188387 24935 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:36.188589 24935 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:36.196310 24935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb2e2fbb25d643a3984af0dcd1bed103: Generated new TSK 0
I20260925 05:14:36.196560 24935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:36.199836 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24845
2026-09-25T05:14:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:36 dist-test-slave-1lvv krb5kdc[24944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:36 dist-test-slave-1lvv krb5kdc[24944](info): set up 2 sockets
Sep 25 05:14:36 dist-test-slave-1lvv krb5kdc[24944](info): commencing operation
krb5kdc: starting...
W20260925 05:14:37.248217 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:37 dist-test-slave-1lvv krb5kdc[24944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:37.379667 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:46453
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:46453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:37.464548 24957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:37.464836 24957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:37.464880 24957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:37.466115 24957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:37.466204 24957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:37.466230 24957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:37.466251 24957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:37.466274 24957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:37.467917 24957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:46453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:37.468226 24957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:37.468480 24957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:37.473671 24963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:37.473694 24962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:37.473695 24965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:37.473985 24957 server_base.cc:1034] running on GCE node
I20260925 05:14:37.474239 24957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:37.474601 24957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:37.475833 24957 hybrid_clock.cc:648] HybridClock initialized: now 1790313277475824 us; error 42 us; skew 500 ppm
Sep 25 05:14:37 dist-test-slave-1lvv krb5kdc[24944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313277, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:37.478413 24957 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:37.478933 24957 webserver.cc:460] Webserver started at http://127.3.6.190:42257/ using document root <none> and password file <none>
I20260925 05:14:37.479141 24957 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:37.479184 24957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:37.479286 24957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:37.480433 24957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "976a39f099584446a330c428abc69332"
format_stamp: "Formatted at 2026-09-25 05:14:37 on dist-test-slave-1lvv"
server_key: "9114efa6b2aff518cf5a814fc92ccbdf"
server_key_iv: "1c7e4088154b4c4e0d1024ac1576ab57"
server_key_version: "encryptionkey@0"
I20260925 05:14:37.480767 24957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "976a39f099584446a330c428abc69332"
format_stamp: "Formatted at 2026-09-25 05:14:37 on dist-test-slave-1lvv"
server_key: "9114efa6b2aff518cf5a814fc92ccbdf"
server_key_iv: "1c7e4088154b4c4e0d1024ac1576ab57"
server_key_version: "encryptionkey@0"
I20260925 05:14:37.482147 24957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:37.482898 24972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:37.483104 24957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:37.483202 24957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "976a39f099584446a330c428abc69332"
format_stamp: "Formatted at 2026-09-25 05:14:37 on dist-test-slave-1lvv"
server_key: "9114efa6b2aff518cf5a814fc92ccbdf"
server_key_iv: "1c7e4088154b4c4e0d1024ac1576ab57"
server_key_version: "encryptionkey@0"
I20260925 05:14:37.483269 24957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:37.512768 24957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:37.515317 24957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:37.515447 24957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:37.519496 24957 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:46453
I20260925 05:14:37.519586 25024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:46453 every 8 connection(s)
I20260925 05:14:37.520056 24957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:37.521015 25025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:37.523427 25025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332: Bootstrap starting.
I20260925 05:14:37.524085 25025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:37.524402 25025 log.cc:824] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:37.525228 25025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332: No bootstrap required, opened a new log
I20260925 05:14:37.526024 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 24957
I20260925 05:14:37.526163 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:37.526355 3098 external_mini_cluster.cc:1428] Setting key bb3ec58c9885df32e570ab65e306e1f5
I20260925 05:14:37.527472 25025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976a39f099584446a330c428abc69332" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46453 } }
I20260925 05:14:37.527679 25025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:37.527724 25025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976a39f099584446a330c428abc69332, State: Initialized, Role: FOLLOWER
I20260925 05:14:37.527966 25025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976a39f099584446a330c428abc69332" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46453 } }
I20260925 05:14:37.528067 25025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:37.528115 25025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:37.528175 25025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:37.529246 25025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976a39f099584446a330c428abc69332" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46453 } }
I20260925 05:14:37.529382 25025 leader_election.cc:302] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 976a39f099584446a330c428abc69332; no voters:
I20260925 05:14:37.529597 25025 leader_election.cc:288] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:37.529647 25030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:37.529795 25030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [term 1 LEADER]: Becoming Leader. State: Replica: 976a39f099584446a330c428abc69332, State: Running, Role: LEADER
I20260925 05:14:37.529875 25025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:37.529927 25030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976a39f099584446a330c428abc69332" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46453 } }
I20260925 05:14:37.530377 25030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976a39f099584446a330c428abc69332. Latest consensus state: current_term: 1 leader_uuid: "976a39f099584446a330c428abc69332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976a39f099584446a330c428abc69332" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46453 } } }
I20260925 05:14:37.530598 25030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:37.530530 25032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "976a39f099584446a330c428abc69332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976a39f099584446a330c428abc69332" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46453 } } }
I20260925 05:14:37.531037 25032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332 [sys.catalog]: This master's current role is: LEADER
W20260925 05:14:37.531944 25046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:37.532014 25046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:37.532127 25039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:37.532320 25039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:37.533883 25039 catalog_manager.cc:1347] Generated new cluster ID: 535301294e67428db7f2311bf86045f9
I20260925 05:14:37.533942 25039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:14:37 dist-test-slave-1lvv krb5kdc[24944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:37.541502 25039 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:37.541517 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:37.526964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45935 (local address 127.3.6.190:46453)
0925 05:14:37.527176 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:37.527184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:37.527212 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:14:37.527419 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:37.527425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:37.527473 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:37.527597 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:37.528513 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:37.529498 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:37.530646 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:37.531340 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:37.531435 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:37.537269 (+ 5834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:37.537275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:37.537275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:37.537291 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:37.539592 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:37.540046 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:37.540052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:37.540052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:37.540171 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:37.540408 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:37.540408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:37.540409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:37.540688 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:37.540891 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:37.540977 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:14:37.541241 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":123,"threads_started":1}
I20260925 05:14:37.542268 25039 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:37.542542 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:37.550478 25039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 976a39f099584446a330c428abc69332: Generated new TSK 0
I20260925 05:14:37.550738 25039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:37.557461 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:37.548226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44519 (local address 127.3.6.190:46453)
0925 05:14:37.548299 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:37.548301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:37.550100 (+ 1799us) server_negotiation.cc:407] Connection header received
0925 05:14:37.550137 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:37.550138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:37.550152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:37.550231 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:37.550986 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:37.551482 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:37.552626 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:37.552852 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:37.552911 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:37.554355 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:37.554356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:37.554356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:37.554364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:37.556293 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:37.556685 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:37.556686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:37.556686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:37.556740 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:37.556966 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:37.556968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:37.556968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:37.557096 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:37.557196 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:37.557324 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:14:37.557391 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:14:37.560029 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 24957
2026-09-25T05:14:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:37 dist-test-slave-1lvv krb5kdc[25056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:37 dist-test-slave-1lvv krb5kdc[25056](info): set up 2 sockets
Sep 25 05:14:37 dist-test-slave-1lvv krb5kdc[25056](info): commencing operation
krb5kdc: starting...
W20260925 05:14:38.600173 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:38 dist-test-slave-1lvv krb5kdc[25056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:38.735524 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45055
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:38.820493 25069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:38.820781 25069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:38.820827 25069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:38.822081 25069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:38.822173 25069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:38.822197 25069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:38.822228 25069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:38.822258 25069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:38.823889 25069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:45055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:45055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:38.824206 25069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:38.824473 25069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:14:38.829849 25069 server_base.cc:1034] running on GCE node
W20260925 05:14:38.829793 25075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:38.829813 25074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:38.829793 25077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:38.830371 25069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:38.830719 25069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:38.831823 25069 hybrid_clock.cc:648] HybridClock initialized: now 1790313278831801 us; error 45 us; skew 500 ppm
Sep 25 05:14:38 dist-test-slave-1lvv krb5kdc[25056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313278, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:38.838426 25069 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:38.838883 25069 webserver.cc:460] Webserver started at http://127.3.6.190:42259/ using document root <none> and password file <none>
I20260925 05:14:38.839092 25069 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:38.839136 25069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:38.839232 25069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:38.840415 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "9623ce7c43bf4c77b042adc55224c8fb"
format_stamp: "Formatted at 2026-09-25 05:14:38 on dist-test-slave-1lvv"
server_key: "738d6bb85544b5ad1a0ca38112288266"
server_key_iv: "cd377789d0797affcd4b0292d70f95ec"
server_key_version: "encryptionkey@0"
I20260925 05:14:38.840761 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "9623ce7c43bf4c77b042adc55224c8fb"
format_stamp: "Formatted at 2026-09-25 05:14:38 on dist-test-slave-1lvv"
server_key: "738d6bb85544b5ad1a0ca38112288266"
server_key_iv: "cd377789d0797affcd4b0292d70f95ec"
server_key_version: "encryptionkey@0"
I20260925 05:14:38.842244 25069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:38.843129 25084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:38.843427 25069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:38.843528 25069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "9623ce7c43bf4c77b042adc55224c8fb"
format_stamp: "Formatted at 2026-09-25 05:14:38 on dist-test-slave-1lvv"
server_key: "738d6bb85544b5ad1a0ca38112288266"
server_key_iv: "cd377789d0797affcd4b0292d70f95ec"
server_key_version: "encryptionkey@0"
I20260925 05:14:38.843592 25069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:38.853261 25069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:38.855186 25069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:38.855319 25069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:38.859815 25069 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:45055
I20260925 05:14:38.859833 25136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:45055 every 8 connection(s)
I20260925 05:14:38.860266 25069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:38.861198 25137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:38.861454 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25069
I20260925 05:14:38.861569 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:38.861820 3098 external_mini_cluster.cc:1428] Setting key 59a741927f6e9f87302689ab3802a84c
I20260925 05:14:38.864692 25137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb: Bootstrap starting.
I20260925 05:14:38.865432 25137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:38.865741 25137 log.cc:824] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:38.866497 25137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb: No bootstrap required, opened a new log
I20260925 05:14:38.867626 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:38.862370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42305 (local address 127.3.6.190:45055)
0925 05:14:38.862565 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:38.862574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:38.862600 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:14:38.863033 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:38.863039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:38.863087 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:38.863223 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:38.863470 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:38.864422 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:38.865576 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:38.866049 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:38.866136 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:38.866892 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:38.866896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:38.866897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:38.867105 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:38.867145 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:38.867161 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:38.867233 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:38.867332 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 05:14:38.867401 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260925 05:14:38.868600 25137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9623ce7c43bf4c77b042adc55224c8fb" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45055 } }
I20260925 05:14:38.868801 25137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:38.868840 25137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9623ce7c43bf4c77b042adc55224c8fb, State: Initialized, Role: FOLLOWER
I20260925 05:14:38.868953 25137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9623ce7c43bf4c77b042adc55224c8fb" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45055 } }
I20260925 05:14:38.869040 25137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:38.869046 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:38.869083 25137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:38.869130 25137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:38.869693 25137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9623ce7c43bf4c77b042adc55224c8fb" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45055 } }
I20260925 05:14:38.869814 25137 leader_election.cc:302] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9623ce7c43bf4c77b042adc55224c8fb; no voters:
I20260925 05:14:38.870013 25137 leader_election.cc:288] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:38.870038 25144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:38.870187 25144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [term 1 LEADER]: Becoming Leader. State: Replica: 9623ce7c43bf4c77b042adc55224c8fb, State: Running, Role: LEADER
I20260925 05:14:38.870272 25144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9623ce7c43bf4c77b042adc55224c8fb" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45055 } }
I20260925 05:14:38.870316 25137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:38.870596 25144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9623ce7c43bf4c77b042adc55224c8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9623ce7c43bf4c77b042adc55224c8fb" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45055 } } }
I20260925 05:14:38.870678 25144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:38.870802 25144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9623ce7c43bf4c77b042adc55224c8fb. Latest consensus state: current_term: 1 leader_uuid: "9623ce7c43bf4c77b042adc55224c8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9623ce7c43bf4c77b042adc55224c8fb" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 45055 } } }
I20260925 05:14:38.870872 25144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:38.871541 25152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:38.872094 25152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:38.873590 25152 catalog_manager.cc:1347] Generated new cluster ID: 744c1a1506684c96bef3943cadd6a48d
I20260925 05:14:38.873643 25152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:38.873803 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:38.870425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48051 (local address 127.3.6.190:45055)
0925 05:14:38.870465 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:38.870466 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:38.870581 (+ 115us) server_negotiation.cc:407] Connection header received
0925 05:14:38.870679 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:38.870680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:38.870691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:38.870750 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:38.870959 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:38.871623 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:38.872498 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:38.872719 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:38.872807 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:38.873543 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:38.873544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:38.873545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:38.873549 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:38.873564 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:38.873569 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:38.873598 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:38.873713 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 05:14:38.873739 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:14:38.898072 25152 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:38.898676 25152 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:38.909315 25152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9623ce7c43bf4c77b042adc55224c8fb: Generated new TSK 0
I20260925 05:14:38.909575 25152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:38.918462 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25069
2026-09-25T05:14:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:38 dist-test-slave-1lvv krb5kdc[25168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:38 dist-test-slave-1lvv krb5kdc[25168](info): set up 2 sockets
Sep 25 05:14:38 dist-test-slave-1lvv krb5kdc[25168](info): commencing operation
krb5kdc: starting...
W20260925 05:14:39.968412 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:40 dist-test-slave-1lvv krb5kdc[25168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:40.098274 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:37679
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:37679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:40.180534 25181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:40.180768 25181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:40.180792 25181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:40.181910 25181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:40.181946 25181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:40.181960 25181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:40.181973 25181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:40.181985 25181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:40.183403 25181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:37679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:37679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:40.183602 25181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:40.183815 25181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:40.188807 25189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:40.188867 25187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:40.188961 25181 server_base.cc:1034] running on GCE node
W20260925 05:14:40.189098 25186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:40.189498 25181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:40.189816 25181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:40.190997 25181 hybrid_clock.cc:648] HybridClock initialized: now 1790313280190990 us; error 38 us; skew 500 ppm
Sep 25 05:14:40 dist-test-slave-1lvv krb5kdc[25168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313280, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:40.193610 25181 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:40.194083 25181 webserver.cc:460] Webserver started at http://127.3.6.190:38837/ using document root <none> and password file <none>
I20260925 05:14:40.194331 25181 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:40.194403 25181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:40.194561 25181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:40.195691 25181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "b965713fdee041bba4d582040ea4aaf4"
format_stamp: "Formatted at 2026-09-25 05:14:40 on dist-test-slave-1lvv"
server_key: "ff8bc97d871b7862eb772cb6bf89f3a0"
server_key_iv: "f974c3a3bc0d073616a62fb8d4411a8e"
server_key_version: "encryptionkey@0"
I20260925 05:14:40.196135 25181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "b965713fdee041bba4d582040ea4aaf4"
format_stamp: "Formatted at 2026-09-25 05:14:40 on dist-test-slave-1lvv"
server_key: "ff8bc97d871b7862eb772cb6bf89f3a0"
server_key_iv: "f974c3a3bc0d073616a62fb8d4411a8e"
server_key_version: "encryptionkey@0"
I20260925 05:14:40.197664 25181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:40.198455 25196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:40.198647 25181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:40.198755 25181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "b965713fdee041bba4d582040ea4aaf4"
format_stamp: "Formatted at 2026-09-25 05:14:40 on dist-test-slave-1lvv"
server_key: "ff8bc97d871b7862eb772cb6bf89f3a0"
server_key_iv: "f974c3a3bc0d073616a62fb8d4411a8e"
server_key_version: "encryptionkey@0"
I20260925 05:14:40.198837 25181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:40.202366 25181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:40.204460 25181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:40.204627 25181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:40.208933 25181 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:37679
I20260925 05:14:40.209012 25248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:37679 every 8 connection(s)
I20260925 05:14:40.209452 25181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:40.210690 25249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:40.213315 25249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4: Bootstrap starting.
I20260925 05:14:40.213586 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25181
I20260925 05:14:40.213678 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:40.213814 3098 external_mini_cluster.cc:1428] Setting key d5a1e357ad315248c15d069c95a3d98a
I20260925 05:14:40.213904 25249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:40.214210 25249 log.cc:824] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:40.214996 25249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4: No bootstrap required, opened a new log
I20260925 05:14:40.216522 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:40.214388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57311 (local address 127.3.6.190:37679)
0925 05:14:40.214589 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:40.214598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:40.214635 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:14:40.215132 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:40.215140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:40.215169 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:40.215324 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:40.216099 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:40.216103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:40.216103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:40.216110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:40.216155 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:40.216166 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:40.216269 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:40.216306 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 05:14:40.216349 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260925 05:14:40.217571 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:40.217509 25249 raft_consensus.cc:348] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965713fdee041bba4d582040ea4aaf4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37679 } }
I20260925 05:14:40.217716 25249 raft_consensus.cc:374] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:14:40.217767 25249 raft_consensus.cc:729] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b965713fdee041bba4d582040ea4aaf4, State: Initialized, Role: FOLLOWER
I20260925 05:14:40.217891 25249 consensus_queue.cc:260] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965713fdee041bba4d582040ea4aaf4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37679 } }
I20260925 05:14:40.217972 25249 raft_consensus.cc:388] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:14:40.218015 25249 raft_consensus.cc:482] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:14:40.218066 25249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:40.218631 25249 raft_consensus.cc:504] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965713fdee041bba4d582040ea4aaf4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37679 } }
I20260925 05:14:40.218745 25249 leader_election.cc:302] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b965713fdee041bba4d582040ea4aaf4; no voters:
I20260925 05:14:40.218923 25249 leader_election.cc:288] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:14:40.218961 25258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:40.219146 25258 raft_consensus.cc:686] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [term 1 LEADER]: Becoming Leader. State: Replica: b965713fdee041bba4d582040ea4aaf4, State: Running, Role: LEADER
I20260925 05:14:40.219340 25258 consensus_queue.cc:237] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965713fdee041bba4d582040ea4aaf4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37679 } }
I20260925 05:14:40.219184 25249 sys_catalog.cc:564] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:40.219856 25259 sys_catalog.cc:455] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b965713fdee041bba4d582040ea4aaf4. Latest consensus state: current_term: 1 leader_uuid: "b965713fdee041bba4d582040ea4aaf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965713fdee041bba4d582040ea4aaf4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37679 } } }
I20260925 05:14:40.219950 25259 sys_catalog.cc:458] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:40.220163 25261 sys_catalog.cc:455] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b965713fdee041bba4d582040ea4aaf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965713fdee041bba4d582040ea4aaf4" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 37679 } } }
I20260925 05:14:40.220391 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:40.219220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50727 (local address 127.3.6.190:37679)
0925 05:14:40.219257 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:40.219259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:40.219269 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:40.219336 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:40.219337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:40.219341 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:40.219388 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:40.220075 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:40.220076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:40.220077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:14:40.220081 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:40.220098 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:14:40.220104 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:40.220140 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:40.220304 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 05:14:40.220320 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:14:40.220551 25261 sys_catalog.cc:458] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:40.220661 25264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:40.221169 25264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:40.222904 25264 catalog_manager.cc:1347] Generated new cluster ID: f5432b93b4044965a1e1c3110a983c69
I20260925 05:14:40.222962 25264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:40.231590 25264 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:40.231817 25264 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:40.235067 25264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b965713fdee041bba4d582040ea4aaf4: Generated new TSK 0
I20260925 05:14:40.235275 25264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:40.241358 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25181
2026-09-25T05:14:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1322 ms)
[----------] 18 tests from AuthTokenIssuingTest (24337 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:40 dist-test-slave-1lvv krb5kdc[25280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:40 dist-test-slave-1lvv krb5kdc[25280](info): set up 2 sockets
Sep 25 05:14:40 dist-test-slave-1lvv krb5kdc[25280](info): commencing operation
krb5kdc: starting...
W20260925 05:14:41.280268 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:41.416213 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:46253
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:46253,127.3.6.189:44815,127.3.6.188:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:41.501109 25293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:14:41.501405 25293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:41.501454 25293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:41.501502 25293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:41.502763 25293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:41.502843 25293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:41.502869 25293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:41.502902 25293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:41.502930 25293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:41.504609 25293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:46253,127.3.6.189:44815,127.3.6.188:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:41.504906 25293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:41.505159 25293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:41.510478 25301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:41.510494 25299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:41.510468 25298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:41.510754 25293 server_base.cc:1034] running on GCE node
I20260925 05:14:41.511114 25293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:41.511451 25293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:41.512625 25293 hybrid_clock.cc:648] HybridClock initialized: now 1790313281512592 us; error 50 us; skew 500 ppm
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:41.515508 25293 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:41.516111 25293 webserver.cc:460] Webserver started at http://127.3.6.190:34303/ using document root <none> and password file <none>
I20260925 05:14:41.516386 25293 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:41.516431 25293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:41.516531 25293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:41.517676 25293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "e791046de03d41dbbb48b78802b5d173"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "8c28a9091059691bff13368cce180f77"
server_key_iv: "7dee7b4dd681311ca67936cb0eeb52b7"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.518041 25293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "e791046de03d41dbbb48b78802b5d173"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "8c28a9091059691bff13368cce180f77"
server_key_iv: "7dee7b4dd681311ca67936cb0eeb52b7"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.519569 25293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:41.520453 25308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:41.520666 25293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:41.520782 25293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "e791046de03d41dbbb48b78802b5d173"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "8c28a9091059691bff13368cce180f77"
server_key_iv: "7dee7b4dd681311ca67936cb0eeb52b7"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.520862 25293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:41.553984 25293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:41.556870 25293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:41.557061 25293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:41.562793 25293 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:46253
I20260925 05:14:41.562915 25360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:46253 every 8 connection(s)
I20260925 05:14:41.563385 25293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:41.564971 25361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:41.567986 25361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:41.572501 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25293
I20260925 05:14:41.572633 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:41.572858 3098 external_mini_cluster.cc:1428] Setting key a60283233a734331d5391ca6e432255d
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:41.581368 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.569260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.3.6.190:46253)
0925 05:14:41.569385 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.569390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.569431 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:14:41.570307 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.570309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.570327 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.570418 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.571373 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.572384 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.573499 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.574005 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.574130 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.577406 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.577418 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.577418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.577428 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.579507 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.580046 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.580050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.580050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.580149 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.580369 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.580370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.580371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.580618 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.580721 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.580918 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.581075 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260925 05:14:41.581372 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.569022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:46253 (local address 127.0.0.1:50150)
0925 05:14:41.569232 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.569291 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.570127 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.570465 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.570469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.570955 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.571256 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.571265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.572533 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.572535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.573393 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.573396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.574113 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.574114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.574181 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.574844 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.574853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.577264 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.579613 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.579620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.579623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.579935 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.580216 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.580216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.580217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.580318 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.580696 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.580701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.580776 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.580891 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.581141 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":94,"thread_start_us":79,"threads_started":1}
I20260925 05:14:41.582295 25361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:41.582752 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.582579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56618)
0925 05:14:41.582645 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.582709 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 05:14:41.582974 25311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.6.189@KRBTEST.COM; defaulting to no policy
I20260925 05:14:41.583472 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.583355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56624)
0925 05:14:41.583399 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.583436 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 05:14:41.583657 25361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:44815: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab.
Entry for principal kudu/127.3.6.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab.
I20260925 05:14:41.603641 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:44815
--webserver_interface=127.3.6.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab
--principal=kudu/127.3.6.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:46253,127.3.6.189:44815,127.3.6.188:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:14:41.632962 25361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } attempt: 1
I20260925 05:14:41.633581 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.633416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56632)
0925 05:14:41.633480 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.633536 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 05:14:41.633960 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.633856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56634)
0925 05:14:41.633900 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.633936 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 05:14:41.634145 25361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:44815: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
W20260925 05:14:41.691262 25368 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:14:41.691568 25368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:41.691587 25368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:41.691608 25368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:41.692874 25368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:41.692927 25368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:41.692941 25368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:41.692955 25368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:41.692967 25368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:41.694603 25368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:46253,127.3.6.189:44815,127.3.6.188:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab
--principal=kudu/127.3.6.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.6.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:41.694902 25368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:41.695156 25368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:41.700260 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:41.700275 25373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:41.700268 25374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:41.700299 25368 server_base.cc:1034] running on GCE node
I20260925 05:14:41.700738 25368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:41.701041 25368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:41.702231 25368 hybrid_clock.cc:648] HybridClock initialized: now 1790313281702227 us; error 44 us; skew 500 ppm
I20260925 05:14:41.702313 25361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } attempt: 2
I20260925 05:14:41.702731 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.702574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56648)
0925 05:14:41.702662 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.702700 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:14:41.703161 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.703035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56654)
0925 05:14:41.703094 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.703125 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 05:14:41.703293 25361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:44815: Network error: Client connection negotiation failed: client connection to 127.3.6.189:44815: connect: Connection refused (error 111)
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:41.705119 25368 init.cc:377] Logged in from keytab as kudu/127.3.6.189@KRBTEST.COM (short username kudu)
I20260925 05:14:41.705525 25368 webserver.cc:460] Webserver started at http://127.3.6.189:38277/ using document root <none> and password file <none>
I20260925 05:14:41.705724 25368 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:41.705766 25368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:41.705858 25368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:41.706903 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data/instance:
uuid: "95db279083fd4c2bbad9c8a4320829d7"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "fcb52fc5e1b98b0dc2bf7b2359a6a89c"
server_key_iv: "c32d4d9871a51f24c604bbddba40fa22"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.707237 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/wal/instance:
uuid: "95db279083fd4c2bbad9c8a4320829d7"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "fcb52fc5e1b98b0dc2bf7b2359a6a89c"
server_key_iv: "c32d4d9871a51f24c604bbddba40fa22"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.708820 25368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:14:41.709646 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:41.709856 25368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:14:41.709977 25368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/wal
uuid: "95db279083fd4c2bbad9c8a4320829d7"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "fcb52fc5e1b98b0dc2bf7b2359a6a89c"
server_key_iv: "c32d4d9871a51f24c604bbddba40fa22"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.710064 25368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:41.728202 25368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:41.730856 25368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:41.731048 25368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:41.735512 25368 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.189:44815
I20260925 05:14:41.735572 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.189:44815 every 8 connection(s)
I20260925 05:14:41.735994 25368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
I20260925 05:14:41.737099 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:41.738785 25436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:41.739493 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25368
I20260925 05:14:41.739653 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-1/wal/instance
I20260925 05:14:41.739876 3098 external_mini_cluster.cc:1428] Setting key d69f05efcb93a127e8955109738c82b6
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.6.188@KRBTEST.COM; defaulting to no policy
I20260925 05:14:41.750350 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.739604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.3.6.190:46253)
0925 05:14:41.739724 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.739726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.739961 (+ 235us) server_negotiation.cc:407] Connection header received
0925 05:14:41.740771 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.740772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.740787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.740851 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.741827 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.742639 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.744216 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.744432 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.744496 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.747291 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.747294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.747294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.747305 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.748841 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.749327 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.749329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.749329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.749393 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.749661 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.749663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.749663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.749792 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.749868 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.750200 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.750279 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 05:14:41.750586 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.739544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:46253 (local address 127.0.0.1:50162)
0925 05:14:41.739748 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.739837 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.740625 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.740929 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.740933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.741349 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.741660 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.741668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.742752 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.742753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.744123 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.744126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.744547 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.744548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.744725 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.745320 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.745331 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.747158 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.748953 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.748961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.748963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.749246 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.749463 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.749464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.749464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.749591 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.749870 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.749878 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.750037 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.750161 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.750342 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
I20260925 05:14:41.751307 25436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.188@KRBTEST.COM" created.
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:14:41.759382 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.751603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56664)
0925 05:14:41.751661 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.751673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.751717 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.752082 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.752083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.752231 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.752292 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.752293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.753151 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.753152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.753680 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.753682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.754183 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.754184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.754251 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.754749 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.754753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.756234 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.758222 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.758223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.758224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.758415 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.758658 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.758659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.758659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.758718 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.759037 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.759038 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.759076 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.759138 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.759258 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:14:41.759457 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.751667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.3.6.189:44815)
0925 05:14:41.751920 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.751924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.751941 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:14:41.751959 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.751960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.751989 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.752083 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.752377 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.753068 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.753778 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.754106 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.754187 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.756338 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.756345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.756346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.756352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.758095 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.758515 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.758519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.758519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.758580 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.758796 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.758797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.758798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.758958 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.759049 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.759167 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.759330 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":223,"thread_start_us":146,"threads_started":1}
I20260925 05:14:41.760085 25436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:41.760501 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.760374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38238)
0925 05:14:41.760425 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.760471 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 05:14:41.760630 25385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
I20260925 05:14:41.761092 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.760996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38244)
0925 05:14:41.761047 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.761071 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 05:14:41.761205 25436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33723: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab.
Entry for principal kudu/127.3.6.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab.
I20260925 05:14:41.771477 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:33723
--webserver_interface=127.3.6.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab
--principal=kudu/127.3.6.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:46253,127.3.6.189:44815,127.3.6.188:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:14:41.794488 25361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } attempt: 3
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:14:41.802117 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.795090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.3.6.189:44815)
0925 05:14:41.795169 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.795171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.795185 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:14:41.795260 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.795261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.795276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.795334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.795706 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.796283 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.797036 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.797269 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.797352 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.799513 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.799515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.799515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.799523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.800928 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.801293 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.801295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.801296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.801356 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.801564 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.801565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.801566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.801690 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.801769 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.801931 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.802016 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:14:41.802117 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.795004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56672)
0925 05:14:41.795101 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.795116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.795190 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.795352 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.795353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.795510 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.795583 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.795585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.796399 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.796400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.796942 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.796944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.797349 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.797350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.797434 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.797941 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.797946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.799410 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.801018 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.801019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.801020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.801201 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.801430 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.801430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.801431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.801497 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.801762 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.801763 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.801813 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.801902 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.802015 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 05:14:41.802583 25361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:41.802958 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.802854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38260)
0925 05:14:41.802904 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.802932 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:14:41.803324 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.803244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38264)
0925 05:14:41.803287 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.803310 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:14:41.803427 25361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33723: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
I20260925 05:14:41.810452 25436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } attempt: 1
I20260925 05:14:41.810993 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.810854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38270)
0925 05:14:41.810917 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.810962 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 05:14:41.811347 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.811245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38284)
0925 05:14:41.811292 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.811317 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 05:14:41.811437 25436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33723: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
I20260925 05:14:41.834589 25361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } attempt: 1
I20260925 05:14:41.835187 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.835030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38288)
0925 05:14:41.835112 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.835159 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 05:14:41.835588 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.835508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38296)
0925 05:14:41.835548 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.835574 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 05:14:41.835698 25361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33723: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
W20260925 05:14:41.860073 25443 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:14:41.860321 25443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:41.860339 25443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:41.860363 25443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:41.861690 25443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:41.861754 25443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:41.861769 25443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:41.861783 25443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:41.861796 25443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:41.863435 25443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:46253,127.3.6.189:44815,127.3.6.188:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab
--principal=kudu/127.3.6.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.6.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:41.863737 25443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:41.864053 25443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:41.869338 25448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:41.869303 25449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:41.869318 25443 server_base.cc:1034] running on GCE node
W20260925 05:14:41.869316 25451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:41.869962 25443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:41.870249 25443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:41.871398 25443 hybrid_clock.cc:648] HybridClock initialized: now 1790313281871386 us; error 31 us; skew 500 ppm
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:41.874001 25443 init.cc:377] Logged in from keytab as kudu/127.3.6.188@KRBTEST.COM (short username kudu)
I20260925 05:14:41.874508 25443 webserver.cc:460] Webserver started at http://127.3.6.188:46253/ using document root <none> and password file <none>
I20260925 05:14:41.874766 25443 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:41.874815 25443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:41.874975 25443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:41.876235 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data/instance:
uuid: "40c59d0b364e4fa491e050c161da8aec"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "17093810883895cb28ea008368b22d5f"
server_key_iv: "e8b1abab751e5f9f3a515291f418dc1d"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.876641 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/wal/instance:
uuid: "40c59d0b364e4fa491e050c161da8aec"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "17093810883895cb28ea008368b22d5f"
server_key_iv: "e8b1abab751e5f9f3a515291f418dc1d"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.878168 25443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 05:14:41.879081 25458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:41.879280 25443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:41.879396 25443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/wal
uuid: "40c59d0b364e4fa491e050c161da8aec"
format_stamp: "Formatted at 2026-09-25 05:14:41 on dist-test-slave-1lvv"
server_key: "17093810883895cb28ea008368b22d5f"
server_key_iv: "e8b1abab751e5f9f3a515291f418dc1d"
server_key_version: "encryptionkey@0"
I20260925 05:14:41.879478 25443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:41.879560 25436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } attempt: 2
I20260925 05:14:41.880000 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.879862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38300)
0925 05:14:41.879928 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.879970 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 05:14:41.880421 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.880335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38310)
0925 05:14:41.880378 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.880405 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 05:14:41.880532 25436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33723: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33723: connect: Connection refused (error 111)
I20260925 05:14:41.901330 25443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:41.903443 25443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:41.903609 25443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:41.908136 25443 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.188:33723
I20260925 05:14:41.908209 25510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.188:33723 every 8 connection(s)
I20260925 05:14:41.908625 25443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
I20260925 05:14:41.909694 25511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:41.910905 25361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } attempt: 2
I20260925 05:14:41.911556 25511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:41.917619 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25443
I20260925 05:14:41.917781 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790313008371521-3098-0/minicluster-data/master-2/wal/instance
I20260925 05:14:41.918040 3098 external_mini_cluster.cc:1428] Setting key 3d23123aa212bfe102c02aa942980775
I20260925 05:14:41.921469 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.911288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38312)
0925 05:14:41.911367 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.911384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.911456 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.912776 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.912777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.912966 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.913004 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.913006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.914247 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.914249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.914807 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.914809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.915406 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.915407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.915460 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.915971 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.915977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.917338 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.920147 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.920148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.920148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.920374 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.920618 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.920619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.920619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.920694 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.921086 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.921087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.921145 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.921241 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.921357 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 05:14:41.921749 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.911624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.3.6.188:33723)
0925 05:14:41.911893 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.911902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.911934 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:14:41.912589 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.912623 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.912649 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.912781 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.913090 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.914174 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.914894 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.915334 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.915417 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.917440 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.917449 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.917449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.917459 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.920061 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.920446 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.920449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.920450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.920518 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.920763 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.920764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.920764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.920983 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.921106 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.921313 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.921447 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260925 05:14:41.922050 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.912414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.3.6.190:46253)
0925 05:14:41.912467 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.912468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.912585 (+ 117us) server_negotiation.cc:407] Connection header received
0925 05:14:41.912726 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.912727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.912737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.912804 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.913512 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.914171 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.915196 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.915383 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.915453 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.918112 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.918114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.918115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.918124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.919690 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.920224 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.920226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.920226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.920294 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.920778 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.920779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.920779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.920979 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.921051 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.921885 (+ 834us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.921964 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:14:41.922417 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.912295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:46253 (local address 127.0.0.1:50172)
0925 05:14:41.912445 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.912493 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.912631 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.912824 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.912826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.913237 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.913391 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.913397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.914295 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.914297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.915113 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.915115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.915501 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.915503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.915574 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.916181 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.916189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.918025 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.919816 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.919824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.919827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.920165 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.920372 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.920373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.920374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.920518 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.921665 (+ 1147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.921672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.921727 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.922183 (+ 456us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.922310 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260925 05:14:41.923943 25361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173: Bootstrap starting.
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:14:41.924324 25511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:41.924832 25361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:41.925225 25361 log.cc:824] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:41.926281 25361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173: No bootstrap required, opened a new log
I20260925 05:14:41.928380 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.918691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.3.6.188:33723)
0925 05:14:41.918852 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.918854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.918860 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:14:41.918877 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.918877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.918885 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.918940 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.919822 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.920489 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.921459 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.921767 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.922356 (+ 589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.924649 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.924651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.924651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.924660 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.926565 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.927020 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.927022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.927022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.927085 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.927343 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.927344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.927344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.927479 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.928069 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.928113 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.928239 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260925 05:14:41.929090 25361 raft_consensus.cc:348] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.929255 25361 raft_consensus.cc:729] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e791046de03d41dbbb48b78802b5d173, State: Initialized, Role: FOLLOWER
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:14:41.929350 25361 consensus_queue.cc:260] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.929693 25518 sys_catalog.cc:455] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.929792 25361 sys_catalog.cc:564] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:41.929913 25518 sys_catalog.cc:458] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:14:41.931994 25529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:41.932056 25529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:41.932896 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.924629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:44815 (local address 127.0.0.1:56674)
0925 05:14:41.924684 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.924694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.924734 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.924888 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.924889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.925396 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.925819 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.925823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.926615 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.926616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.927132 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.927134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.927564 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.927565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.927642 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.928129 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.928135 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.929623 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.931591 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.931592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.931592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.931857 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.932162 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.932163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.932163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.932248 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.932535 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.932535 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.932585 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.932650 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.932758 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 05:14:41.933573 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.924718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.3.6.189:44815)
0925 05:14:41.924769 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.924772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.924780 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:41.924806 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.924807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.924817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.924872 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.925952 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.926525 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.927246 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.927477 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.927554 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.929740 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.929742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.929743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.929751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.931494 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.931943 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.931944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.931945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.932006 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.932317 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.932318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.932318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.932450 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.932509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.933372 (+ 863us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.933462 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 05:14:41.933955 25511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:41.932819 25293 master_runner.cc:186] Error getting master registration for 127.3.6.189:44815: OK, instance_id { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" instance_seqno: 1790313281734245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:14:41.935432 25293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:14:41.938795 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.929098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.3.6.190:46253)
0925 05:14:41.929140 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.929142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.929150 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:41.929218 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.929219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.929228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.929276 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.929940 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.931164 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.932271 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.932548 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.932610 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.935841 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.935842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.935843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.935851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.937601 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.937977 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.937978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.937979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.938041 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.938267 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.938268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.938268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.938388 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.938563 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.938603 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.938696 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:14:41.942238 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.934229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38326)
0925 05:14:41.934271 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.934280 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.934315 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.934462 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.934463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.934624 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.934669 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.934671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.935642 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.935644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.936496 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.936500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.937066 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.937066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.937119 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.937514 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.937519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.939002 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.941128 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.941129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.941129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.941345 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.941533 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.941533 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.941533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.941582 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.941935 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.941936 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.941987 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.942060 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.942168 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 05:14:41.942647 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.934281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.3.6.188:33723)
0925 05:14:41.934320 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.934322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.934332 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:41.934356 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.934356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.934367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.934439 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.934753 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.935504 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.936595 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.936937 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.937015 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.939098 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.939100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.939100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.939109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.940997 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.941423 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.941424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.941424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.941473 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.941654 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.941655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.941655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.941844 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.941920 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.942459 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.942549 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:14:41.944149 25511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: Bootstrap starting.
I20260925 05:14:41.944928 25511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: Neither blocks nor log segments found. Creating new log.
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:14:41.945292 25511 log.cc:824] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:41.946202 25511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: No bootstrap required, opened a new log
I20260925 05:14:41.948601 25511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.948776 25511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c59d0b364e4fa491e050c161da8aec, State: Initialized, Role: FOLLOWER
I20260925 05:14:41.948872 25511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.949211 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.939683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.3.6.189:44815)
0925 05:14:41.939728 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.939730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.939738 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:41.939859 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.939860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.939869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.939926 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.940535 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.941008 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.941715 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.941976 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.942051 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.945913 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.945916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.945916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.945925 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.947803 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.948216 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.948218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.948218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.948288 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.948506 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.948507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.948507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.948661 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.948936 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.949046 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.949127 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:14:41.949215 25511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:41.951043 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 05:14:41.951453 25543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:41.951498 25543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:41.951671 25532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.951733 25532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:41.952235 25335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e791046de03d41dbbb48b78802b5d173"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50174
I20260925 05:14:41.952414 25335 raft_consensus.cc:482] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:14:41.952473 25335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:41.953415 25335 raft_consensus.cc:504] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.953697 25335 leader_election.cc:288] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [CANDIDATE]: Term 1 election: Requested vote from peers 95db279083fd4c2bbad9c8a4320829d7 (127.3.6.189:44815), 40c59d0b364e4fa491e050c161da8aec (127.3.6.188:33723)
I20260925 05:14:41.953994 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e791046de03d41dbbb48b78802b5d173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "40c59d0b364e4fa491e050c161da8aec"
I20260925 05:14:41.954098 25485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:41.954977 25485 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e791046de03d41dbbb48b78802b5d173 in term 1.
I20260925 05:14:41.955183 25310 leader_election.cc:302] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c59d0b364e4fa491e050c161da8aec, e791046de03d41dbbb48b78802b5d173; no voters:
I20260925 05:14:41.955314 25518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:41.955499 25518 raft_consensus.cc:686] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [term 1 LEADER]: Becoming Leader. State: Replica: e791046de03d41dbbb48b78802b5d173, State: Running, Role: LEADER
I20260925 05:14:41.955590 25518 consensus_queue.cc:237] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.956090 25518 sys_catalog.cc:455] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e791046de03d41dbbb48b78802b5d173. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.956187 25518 sys_catalog.cc:458] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:41.954499 25443 master_runner.cc:186] Error getting master registration for 127.3.6.189:44815: OK, instance_id { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" instance_seqno: 1790313281734245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:14:41.956809 25443 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:14:41.956729 25551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:41.957541 25551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:41.959652 25485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [term 1 FOLLOWER]: Refusing update from remote peer e791046de03d41dbbb48b78802b5d173: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:41.960146 25518 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:41.962545 25532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e791046de03d41dbbb48b78802b5d173. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.962690 25532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:41.970013 25545 sys_catalog.cc:455] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.970180 25545 sys_catalog.cc:458] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:41.970685 25532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.970781 25532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:41.971040 25551 catalog_manager.cc:1347] Generated new cluster ID: 5f2b3d0d5ef143eea137753e08767718
I20260925 05:14:41.971092 25551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:41.971653 25436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } attempt: 3
I20260925 05:14:41.975845 25552 mvcc.cc:205] Tried to move back new op lower bound from 7333123202903105536 to 7333123202890948608. Current Snapshot: MvccSnapshot[applied={T|T < 7333123202903105536 or (T in {7333123202903105536})}]
Sep 25 05:14:41 dist-test-slave-1lvv krb5kdc[25280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313281, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:14:41.979899 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.972022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33723 (local address 127.0.0.1:38346)
0925 05:14:41.972075 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:41.972088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:41.972141 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:41.972298 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:41.972299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:41.972494 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:41.972540 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.972542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.973430 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.973432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.974237 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:41.974240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.974768 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.974770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:41.974833 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.975348 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:41.975354 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:41.976924 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:41.978892 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.978893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.978894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.979095 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.979276 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:41.979277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:41.979277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:41.979327 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:41.979563 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:41.979564 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:41.979607 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 05:14:41.979663 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:14:41.979797 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:14:41.980258 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.972128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.3.6.188:33723)
0925 05:14:41.972184 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.972187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.972194 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:14:41.972216 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.972217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.972226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.972282 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.972609 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.973353 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.974317 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.974648 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.974710 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.977031 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.977033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.977033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.977041 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.978819 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.979163 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.979163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.979164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.979215 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.979380 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.979380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.979381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.979495 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.979543 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.980089 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.980169 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:14:41.981420 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.960810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.3.6.188:33723)
0925 05:14:41.960846 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.960848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.968103 (+ 7255us) server_negotiation.cc:407] Connection header received
0925 05:14:41.968226 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.968227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.968241 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.968297 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.972500 (+ 4203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.973232 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.974226 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.974523 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.974564 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.976152 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.976154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.976154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.976161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.978445 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.980653 (+ 2208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.980654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.980654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.980713 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.980888 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.980888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.980888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.981004 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.981090 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.981267 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.981339 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:14:41.982007 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: Bootstrap starting.
I20260925 05:14:41.982313 25556 mvcc.cc:205] Tried to move back new op lower bound from 7333123202903105536 to 7333123202890948608. Current Snapshot: MvccSnapshot[applied={T|T < 7333123202903105536}]
I20260925 05:14:41.982573 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.958636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.3.6.189:44815)
0925 05:14:41.958687 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.958689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.959160 (+ 471us) server_negotiation.cc:407] Connection header received
0925 05:14:41.959647 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.959648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.959662 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.959727 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.961434 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.962200 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.963825 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.964722 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.964810 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.966520 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.966522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.966522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.966531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.968367 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.968864 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.968865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.968865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.968928 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.969124 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.969125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.969126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.969275 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.969386 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.982040 (+ 12654us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.982293 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:14:41.982888 25436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:41.983347 25436 log.cc:824] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:41.984575 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: No bootstrap required, opened a new log
I20260925 05:14:41.985845 25551 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:41.986595 25551 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:41.986914 25436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.987085 25436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95db279083fd4c2bbad9c8a4320829d7, State: Initialized, Role: FOLLOWER
I20260925 05:14:41.987206 25436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } }
I20260925 05:14:41.987587 25436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:41.987836 25410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e791046de03d41dbbb48b78802b5d173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95db279083fd4c2bbad9c8a4320829d7"
I20260925 05:14:41.987942 25410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:41.988978 25410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e791046de03d41dbbb48b78802b5d173 in term 1.
I20260925 05:14:41.989189 25409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [term 1 FOLLOWER]: Refusing update from remote peer e791046de03d41dbbb48b78802b5d173: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:41.989434 25545 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:41.990715 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:41.960288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.3.6.190:46253)
0925 05:14:41.960336 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:41.960339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:41.964350 (+ 4011us) server_negotiation.cc:407] Connection header received
0925 05:14:41.967909 (+ 3559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:41.967911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:41.967928 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:41.967997 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:41.977021 (+ 9024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.977742 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.983228 (+ 5486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:41.983508 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:41.983551 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:41.985872 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:41.985873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:41.985874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:41.985881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:41.989372 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.989817 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.989818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.989818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.989879 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:41.990071 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:41.990072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:41.990072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:41.990200 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:41.990293 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:41.990504 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:14:41.990608 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:14:41.992231 25562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.992553 25562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:41.992645 25562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e791046de03d41dbbb48b78802b5d173. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.992717 25562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:41.992758 25562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.992820 25562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:41.995559 25551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173: Generated new TSK 0
I20260925 05:14:41.995796 25551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:14:41.996502 25518 sys_catalog.cc:455] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e791046de03d41dbbb48b78802b5d173" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e791046de03d41dbbb48b78802b5d173" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 46253 } } peers { permanent_uuid: "95db279083fd4c2bbad9c8a4320829d7" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 44815 } } peers { permanent_uuid: "40c59d0b364e4fa491e050c161da8aec" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33723 } } }
I20260925 05:14:41.996608 25518 sys_catalog.cc:458] T 00000000000000000000000000000000 P e791046de03d41dbbb48b78802b5d173 [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:41.997072 25574 mvcc.cc:205] Tried to move back new op lower bound from 7333123203008598016 to 7333123202890948608. Current Snapshot: MvccSnapshot[applied={T|T < 7333123202903105536}]
W20260925 05:14:41.997819 25577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:41.997925 25577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 05:14:42.045735 25364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.044151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.3.6.190:46253)
0925 05:14:42.044201 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.044204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.044926 (+ 722us) server_negotiation.cc:407] Connection header received
0925 05:14:42.045044 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.045045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.045058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.045093 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.045655 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50200: BlockingRecv error: recv got EOF from 127.0.0.1:50200 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 05:14:42.046607 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.044151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.3.6.189:44815)
0925 05:14:42.044201 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.044204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.046012 (+ 1808us) server_negotiation.cc:407] Connection header received
0925 05:14:42.046120 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.046121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.046145 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.046179 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.046539 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56704: BlockingRecv error: recv got EOF from 127.0.0.1:56704 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 05:14:42.047472 25516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.044591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.3.6.188:33723)
0925 05:14:42.044646 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.044648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.046836 (+ 2188us) server_negotiation.cc:407] Connection header received
0925 05:14:42.046987 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.046988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.047000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.047032 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.047406 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38356: BlockingRecv error: recv got EOF from 127.0.0.1:38356 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 05:14:42.050199 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.049335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.3.6.189:44815)
0925 05:14:42.049373 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.049375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.049613 (+ 238us) server_negotiation.cc:407] Connection header received
0925 05:14:42.049733 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.049734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.049756 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.049789 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.050150 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56712: BlockingRecv error: recv got EOF from 127.0.0.1:56712 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 05:14:42.053383 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.052586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.3.6.189:44815)
0925 05:14:42.052619 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.052621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.052876 (+ 255us) server_negotiation.cc:407] Connection header received
0925 05:14:42.052968 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.052969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.052979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.053008 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.053334 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56724: BlockingRecv error: recv got EOF from 127.0.0.1:56724 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 05:14:42.057605 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.056681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.3.6.189:44815)
0925 05:14:42.056732 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.056734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.057037 (+ 303us) server_negotiation.cc:407] Connection header received
0925 05:14:42.057155 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.057156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.057168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.057199 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.057553 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56732: BlockingRecv error: recv got EOF from 127.0.0.1:56732 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 05:14:42.064186 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.063051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.3.6.189:44815)
0925 05:14:42.063098 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.063100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.063438 (+ 338us) server_negotiation.cc:407] Connection header received
0925 05:14:42.063553 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.063554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.063568 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.063602 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.064111 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56746: BlockingRecv error: recv got EOF from 127.0.0.1:56746 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 05:14:42.074671 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.073581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.3.6.189:44815)
0925 05:14:42.073628 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.073630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.073985 (+ 355us) server_negotiation.cc:407] Connection header received
0925 05:14:42.074101 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.074102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.074115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.074162 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.074606 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56758: BlockingRecv error: recv got EOF from 127.0.0.1:56758 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 05:14:42.093284 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.092247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.3.6.189:44815)
0925 05:14:42.092294 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.092297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.092628 (+ 331us) server_negotiation.cc:407] Connection header received
0925 05:14:42.092742 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.092744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.092757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.092789 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.093212 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56766: BlockingRecv error: recv got EOF from 127.0.0.1:56766 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 05:14:42.128543 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.126810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.3.6.189:44815)
0925 05:14:42.126861 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.126864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.127188 (+ 324us) server_negotiation.cc:407] Connection header received
0925 05:14:42.127916 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.127917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.127935 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.127973 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.128458 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56774: BlockingRecv error: recv got EOF from 127.0.0.1:56774 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 05:14:42.194926 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.194056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.3.6.189:44815)
0925 05:14:42.194163 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.194165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.194244 (+ 79us) server_negotiation.cc:407] Connection header received
0925 05:14:42.194349 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.194350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.194362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.194443 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.194861 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56780: BlockingRecv error: recv got EOF from 127.0.0.1:56780 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 05:14:42.325201 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.324352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.3.6.189:44815)
0925 05:14:42.324416 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.324419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.324534 (+ 115us) server_negotiation.cc:407] Connection header received
0925 05:14:42.324664 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.324665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.324683 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.324756 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.325129 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56794: BlockingRecv error: recv got EOF from 127.0.0.1:56794 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 05:14:42.564531 25356 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.3.6.190:46253
W20260925 05:14:42.583359 25440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:42.582608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.3.6.189:44815)
0925 05:14:42.582679 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:42.582681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:42.582691 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:42.582753 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:42.582755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:42.582766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:42.582820 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:42.583260 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56810: BlockingRecv error: recv got EOF from 127.0.0.1:56810 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:14:42.952030 25543 catalog_manager.cc:1259] Loaded cluster ID: 5f2b3d0d5ef143eea137753e08767718
I20260925 05:14:42.952139 25543 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: loading cluster ID for follower catalog manager: success
I20260925 05:14:42.953014 25543 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: acquiring CA information for follower catalog manager: success
I20260925 05:14:42.953380 25543 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40c59d0b364e4fa491e050c161da8aec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:14:42.998349 25577 catalog_manager.cc:1259] Loaded cluster ID: 5f2b3d0d5ef143eea137753e08767718
I20260925 05:14:42.998431 25577 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: loading cluster ID for follower catalog manager: success
I20260925 05:14:42.999289 25577 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: acquiring CA information for follower catalog manager: success
I20260925 05:14:42.999656 25577 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95db279083fd4c2bbad9c8a4320829d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:14:43.099164 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:43.096518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.3.6.189:44815)
0925 05:14:43.096624 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:43.096625 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:43.096636 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:14:43.096694 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:43.096694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:43.096745 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:43.096846 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:43.097073 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.097579 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.098391 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.098591 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.098674 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:43.098817 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:43.098933 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:43.098992 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:43.099025 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 05:14:43.099044 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260925 05:14:43.103583 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:43.101087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.3.6.188:33723)
0925 05:14:43.101304 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:43.101306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:43.101314 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:43.101352 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:43.101353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:43.101387 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:43.101462 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:43.101605 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.102098 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.102804 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.103000 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.103052 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:43.103226 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:43.103334 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:43.103393 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:43.103498 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 05:14:43.103515 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":169,"threads_started":1}
I20260925 05:14:43.108392 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:43.105678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.3.6.188:33723)
0925 05:14:43.105717 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:43.105719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:43.105727 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:43.105788 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:43.105789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:43.105798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:43.105848 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:43.105994 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.106745 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.107564 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.107867 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.107934 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:43.108082 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:43.108120 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:43.108171 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:43.108274 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 05:14:43.108300 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:14:43.108623 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:43.105284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.3.6.189:44815)
0925 05:14:43.105341 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:43.105344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:43.105561 (+ 217us) server_negotiation.cc:407] Connection header received
0925 05:14:43.105598 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:43.105599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:43.105613 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:43.105669 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:43.105832 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.106611 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.107714 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.108063 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.108148 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:43.108313 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:43.108352 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:43.108385 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:43.108493 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 05:14:43.108527 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:14:43.109130 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:43.105282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.3.6.190:46253)
0925 05:14:43.105716 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:43.105718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:43.105728 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:43.105775 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:43.105776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:43.105832 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:43.105915 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:43.106049 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.106537 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.107821 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:43.108135 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:43.108194 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:43.108432 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:43.108548 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:43.108602 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:43.109014 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 05:14:43.109043 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":61,"threads_started":1}
I20260925 05:14:43.110759 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25293
I20260925 05:14:43.116946 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25368
I20260925 05:14:43.122325 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25443
2026-09-25T05:14:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2882 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:14:43 dist-test-slave-1lvv krb5kdc[25663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:14:43 dist-test-slave-1lvv krb5kdc[25663](info): set up 2 sockets
Sep 25 05:14:43 dist-test-slave-1lvv krb5kdc[25663](info): commencing operation
krb5kdc: starting...
W20260925 05:14:44.176524 3098 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/test-user.keytab.
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.6.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
Entry for principal kudu/127.3.6.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab.
I20260925 05:14:44.310788 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:42549
--webserver_interface=127.3.6.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:42549,127.3.6.189:43597,127.3.6.188:33051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:14:44.395900 25676 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:14:44.396142 25676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:44.396160 25676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:44.396183 25676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:44.397425 25676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:44.397478 25676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:44.397493 25676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:44.397531 25676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:44.397559 25676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:44.399154 25676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:42549,127.3.6.189:43597,127.3.6.188:33051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.190.keytab
--principal=kudu/127.3.6.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.190:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.6.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:44.399428 25676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:44.399680 25676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:44.404659 25684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:44.404667 25682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:44.404667 25681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:44.405066 25676 server_base.cc:1034] running on GCE node
I20260925 05:14:44.405332 25676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:44.405640 25676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:44.406834 25676 hybrid_clock.cc:648] HybridClock initialized: now 1790313284406817 us; error 40 us; skew 500 ppm
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:44.429984 25676 init.cc:377] Logged in from keytab as kudu/127.3.6.190@KRBTEST.COM (short username kudu)
I20260925 05:14:44.430550 25676 webserver.cc:460] Webserver started at http://127.3.6.190:39117/ using document root <none> and password file <none>
I20260925 05:14:44.430821 25676 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:44.430893 25676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:44.431051 25676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:44.432487 25676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data/instance:
uuid: "737a3ae773a04da6b13ef837e998384d"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "5e9122206f0cb3c01a1f44e5842df3bd"
server_key_iv: "e1eda13eceab8081b815c27c76a84b2c"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.432965 25676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/wal/instance:
uuid: "737a3ae773a04da6b13ef837e998384d"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "5e9122206f0cb3c01a1f44e5842df3bd"
server_key_iv: "e1eda13eceab8081b815c27c76a84b2c"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.434444 25676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:44.435266 25691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:44.435434 25676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:44.435541 25676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/wal
uuid: "737a3ae773a04da6b13ef837e998384d"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "5e9122206f0cb3c01a1f44e5842df3bd"
server_key_iv: "e1eda13eceab8081b815c27c76a84b2c"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.435628 25676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:44.450410 25676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:44.452880 25676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:44.453063 25676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:44.457559 25676 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.190:42549
I20260925 05:14:44.457594 25743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.190:42549 every 8 connection(s)
I20260925 05:14:44.458014 25676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/data/info.pb
I20260925 05:14:44.459074 25744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:44.460762 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:44.466718 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25676
I20260925 05:14:44.466795 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-0/wal/instance
I20260925 05:14:44.467016 3098 external_mini_cluster.cc:1428] Setting key 74bb080a452699ea30356ecfae07d997
I20260925 05:14:44.471084 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.461447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:42549 (local address 127.0.0.1:58350)
0925 05:14:44.461619 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.461683 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.462160 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.462414 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.462416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.462698 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.462866 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.462870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.463937 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.463938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.464626 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.464627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.465129 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.465130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.465186 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.465679 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.465688 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.467193 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.469340 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.469348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.469351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.469654 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.469901 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.469902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.469902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.470021 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.470431 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.470437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.470506 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.470598 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.470855 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":121,"thread_start_us":75,"threads_started":1}
I20260925 05:14:44.471083 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.461499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.3.6.190:42549)
0925 05:14:44.461685 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.461692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.461795 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:14:44.462288 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.462290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.462308 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.462390 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.462950 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.463845 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.464705 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.465035 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.465123 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.467292 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.467301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.467302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.467310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.469255 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.469743 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.469745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.469745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.469828 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.470106 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.470107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.470107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.470356 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.470456 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.470636 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.470809 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260925 05:14:44.471978 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:44.472349 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.472224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32926)
0925 05:14:44.472271 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.472322 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:14:44.472500 25694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
I20260925 05:14:44.472935 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.472844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32936)
0925 05:14:44.472889 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.472913 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:14:44.473038 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:43597: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.6.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab.
Entry for principal kudu/127.3.6.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab.
I20260925 05:14:44.494699 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:43597
--webserver_interface=127.3.6.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab
--principal=kudu/127.3.6.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:42549,127.3.6.189:43597,127.3.6.188:33051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:14:44.522274 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } attempt: 1
I20260925 05:14:44.522831 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.522680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32942)
0925 05:14:44.522741 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.522801 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:14:44.523231 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.523134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32948)
0925 05:14:44.523184 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.523213 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 05:14:44.523322 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:43597: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
W20260925 05:14:44.579838 25751 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:14:44.580119 25751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:44.580150 25751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:44.580222 25751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:44.581475 25751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:44.581552 25751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:44.581576 25751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:44.581597 25751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:44.581630 25751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:44.583216 25751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:42549,127.3.6.189:43597,127.3.6.188:33051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.189.keytab
--principal=kudu/127.3.6.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.189:43597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.6.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:44.583519 25751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:44.583791 25751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:44.588809 25757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:44.588804 25759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:44.589139 25751 server_base.cc:1034] running on GCE node
W20260925 05:14:44.589211 25756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:44.589486 25751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:44.589824 25751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:44.590986 25751 hybrid_clock.cc:648] HybridClock initialized: now 1790313284590975 us; error 35 us; skew 500 ppm
I20260925 05:14:44.591454 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } attempt: 2
I20260925 05:14:44.591899 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.591725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32960)
0925 05:14:44.591809 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.591844 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 05:14:44.592312 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.592198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32968)
0925 05:14:44.592251 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.592284 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 05:14:44.592417 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.189:43597: Network error: Client connection negotiation failed: client connection to 127.3.6.189:43597: connect: Connection refused (error 111)
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:44.594015 25751 init.cc:377] Logged in from keytab as kudu/127.3.6.189@KRBTEST.COM (short username kudu)
I20260925 05:14:44.594540 25751 webserver.cc:460] Webserver started at http://127.3.6.189:43909/ using document root <none> and password file <none>
I20260925 05:14:44.594789 25751 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:44.594836 25751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:44.595009 25751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:44.596202 25751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data/instance:
uuid: "872f3ae001164b95ae84810b2be92793"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "8812db72944d1dcac6a1afe0eae37576"
server_key_iv: "408eb37044a080e19ebd1f086c4d1b1a"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.596582 25751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/wal/instance:
uuid: "872f3ae001164b95ae84810b2be92793"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "8812db72944d1dcac6a1afe0eae37576"
server_key_iv: "408eb37044a080e19ebd1f086c4d1b1a"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.597993 25751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:14:44.598762 25766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:44.598937 25751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:14:44.599031 25751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/wal
uuid: "872f3ae001164b95ae84810b2be92793"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "8812db72944d1dcac6a1afe0eae37576"
server_key_iv: "408eb37044a080e19ebd1f086c4d1b1a"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.599103 25751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:44.629695 25751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:44.632333 25751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:44.632508 25751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:44.636902 25751 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.189:43597
I20260925 05:14:44.636943 25818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.189:43597 every 8 connection(s)
I20260925 05:14:44.637323 25751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/data/info.pb
I20260925 05:14:44.638549 25819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:44.640538 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25751
I20260925 05:14:44.640342 25819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:44.640640 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-1/wal/instance
I20260925 05:14:44.640836 3098 external_mini_cluster.cc:1428] Setting key a238f158be6737e0ec8b85cac0c95f5c
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.6.188@KRBTEST.COM; defaulting to no policy
I20260925 05:14:44.650660 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.641286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.3.6.190:42549)
0925 05:14:44.641364 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.641367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.641627 (+ 260us) server_negotiation.cc:407] Connection header received
0925 05:14:44.642123 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.642124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.642138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.642226 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.643010 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.643515 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.644645 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.644921 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.644987 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.647566 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.647568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.647568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.647577 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.649111 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.649592 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.649594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.649594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.649664 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.649934 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.649935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.649935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.650060 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.650134 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.650460 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.650552 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:14:44.650848 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.641263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:42549 (local address 127.0.0.1:58358)
0925 05:14:44.641477 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.641524 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.642026 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.642323 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.642327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.642690 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.642905 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.642910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.643653 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.643654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.644537 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.644540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.645017 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.645019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.645149 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.645760 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.645768 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.647449 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.649215 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.649224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.649227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.649515 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.649732 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.649733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.649733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.649859 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.650126 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.650133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.650325 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.650440 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.650608 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":123,"threads_started":1}
I20260925 05:14:44.651521 25819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.6.188@KRBTEST.COM" created.
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:14:44.661187 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.651866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32980)
0925 05:14:44.651951 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.651964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.652030 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.653305 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.653306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.653456 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.653510 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.653511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.654595 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.654596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.655110 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.655112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.655638 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.655639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.655717 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.656277 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.656282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.657901 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.660034 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.660035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.660036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.660223 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.660446 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.660447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.660447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.660504 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.660817 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.660818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.660859 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.660942 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.661066 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:14:44.661289 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.651917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.3.6.189:43597)
0925 05:14:44.653146 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.653150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.653170 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:14:44.653190 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.653192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.653204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.653292 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.653602 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.654491 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.655206 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.655545 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.655638 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.658013 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.658026 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.658026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.658038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.659932 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.660320 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.660324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.660325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.660380 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.660564 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.660565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.660566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.660741 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.660830 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.660977 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.661159 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":1198,"thread_start_us":139,"threads_started":1}
I20260925 05:14:44.662145 25819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:44.662624 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.662474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47326)
0925 05:14:44.662544 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.662597 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:14:44.662794 25769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
I20260925 05:14:44.663249 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.663164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47340)
0925 05:14:44.663204 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.663230 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 05:14:44.663385 25819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33051: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.6.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab.
Entry for principal kudu/127.3.6.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab.
I20260925 05:14:44.672051 3098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH05q1Z/build/release/bin/kudu
/tmp/dist-test-taskH05q1Z/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:33051
--webserver_interface=127.3.6.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab
--principal=kudu/127.3.6.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.6.148:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.6.190:42549,127.3.6.189:43597,127.3.6.188:33051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:14:44.683565 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } attempt: 3
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:14:44.691217 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.684030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:32996)
0925 05:14:44.684122 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.684134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.684179 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.684341 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.684342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.684537 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.684638 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.684640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.685297 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.685298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.686152 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.686154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.686546 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.686547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.686626 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.687090 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.687095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.688525 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.690176 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.690177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.690178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.690361 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.690577 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.690578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.690578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.690640 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.690934 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.690936 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.690984 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.691048 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.691141 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:14:44.691605 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.684099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.3.6.189:43597)
0925 05:14:44.684157 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.684158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.684170 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:44.684243 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.684244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.684253 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.684319 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.684727 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.685219 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.686234 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.686471 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.686531 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.688618 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.688620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.688620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.688628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.690072 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.690446 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.690447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.690448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.690506 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.690701 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.690702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.690702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.690847 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.691377 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.691429 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.691514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 05:14:44.692098 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:44.692457 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.692371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47350)
0925 05:14:44.692407 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.692439 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 05:14:44.692751 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.692674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47356)
0925 05:14:44.692721 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.692740 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 05:14:44.692883 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33051: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
I20260925 05:14:44.712682 25819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } attempt: 1
I20260925 05:14:44.713315 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.713155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47360)
0925 05:14:44.713228 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.713279 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:14:44.713873 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.713710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47376)
0925 05:14:44.713780 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.713839 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:14:44.713977 25819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33051: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
I20260925 05:14:44.724030 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } attempt: 1
I20260925 05:14:44.724560 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.724411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47388)
0925 05:14:44.724486 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.724530 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 05:14:44.725014 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.724868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47392)
0925 05:14:44.724932 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.724981 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 05:14:44.725147 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33051: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
W20260925 05:14:44.759258 25826 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:14:44.759511 25826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:14:44.759531 25826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:14:44.759552 25826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:14:44.760895 25826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:14:44.760954 25826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:14:44.760969 25826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:14:44.760982 25826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:14:44.760995 25826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:14:44.763110 25826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.6.148:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.6.190:42549,127.3.6.189:43597,127.3.6.188:33051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/krb5kdc/kudu_127.3.6.188.keytab
--principal=kudu/127.3.6.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.6.188:33051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.6.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:14:44.763420 25826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:14:44.763633 25826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:14:44.768684 25831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:44.768684 25834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:14:44.768771 25832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:14:44.769001 25826 server_base.cc:1034] running on GCE node
I20260925 05:14:44.769279 25826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:14:44.769613 25826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:14:44.770804 25826 hybrid_clock.cc:648] HybridClock initialized: now 1790313284770786 us; error 47 us; skew 500 ppm
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:14:44.773420 25826 init.cc:377] Logged in from keytab as kudu/127.3.6.188@KRBTEST.COM (short username kudu)
I20260925 05:14:44.773878 25826 webserver.cc:460] Webserver started at http://127.3.6.188:35187/ using document root <none> and password file <none>
I20260925 05:14:44.774092 25826 fs_manager.cc:359] Metadata directory not provided
I20260925 05:14:44.774135 25826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:14:44.774243 25826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:14:44.775359 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data/instance:
uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "b46ba663f9d756377c56ed2458c87468"
server_key_iv: "a061ddb547107f25ab7be8e7edbfcfa6"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.775710 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/wal/instance:
uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "b46ba663f9d756377c56ed2458c87468"
server_key_iv: "a061ddb547107f25ab7be8e7edbfcfa6"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.777302 25826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:44.778206 25841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:14:44.778396 25826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:14:44.778491 25826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data,/tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/wal
uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2"
format_stamp: "Formatted at 2026-09-25 05:14:44 on dist-test-slave-1lvv"
server_key: "b46ba663f9d756377c56ed2458c87468"
server_key_iv: "a061ddb547107f25ab7be8e7edbfcfa6"
server_key_version: "encryptionkey@0"
I20260925 05:14:44.778576 25826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:14:44.782126 25819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } attempt: 2
I20260925 05:14:44.782554 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.782424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47398)
0925 05:14:44.782480 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.782523 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 05:14:44.782872 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.782808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47402)
0925 05:14:44.782832 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.782856 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 05:14:44.782967 25819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33051: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
I20260925 05:14:44.797649 25826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:14:44.799674 25826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:14:44.799880 25826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:14:44.800312 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } attempt: 2
I20260925 05:14:44.800738 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.800613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47416)
0925 05:14:44.800669 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.800702 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:14:44.801080 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.800993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47430)
0925 05:14:44.801036 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.801060 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 05:14:44.801186 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.6.188:33051: Network error: Client connection negotiation failed: client connection to 127.3.6.188:33051: connect: Connection refused (error 111)
I20260925 05:14:44.804458 25826 rpc_server.cc:307] RPC server started. Bound to: 127.3.6.188:33051
I20260925 05:14:44.804493 25893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.6.188:33051 every 8 connection(s)
I20260925 05:14:44.804873 25826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/data/info.pb
I20260925 05:14:44.805959 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:14:44.807679 25894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:44.808343 3098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH05q1Z/build/release/bin/kudu as pid 25826
I20260925 05:14:44.808458 3098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH05q1Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790313008371521-3098-0/minicluster-data/master-2/wal/instance
I20260925 05:14:44.808736 3098 external_mini_cluster.cc:1428] Setting key 9e418c49d3fd7c1d567cc70e72e25e42
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:14:44.818524 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.808598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.3.6.190:42549)
0925 05:14:44.808657 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.808658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.809028 (+ 370us) server_negotiation.cc:407] Connection header received
0925 05:14:44.809788 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.809789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.809803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.809883 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.810431 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.810934 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.812138 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.812444 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.812570 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.815120 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.815122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.815122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.815130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.816946 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.817470 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.817472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.817473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.817533 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.817816 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.817817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.817817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.817943 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.818012 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.818337 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.818424 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:14:44.818730 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.808590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.190:42549 (local address 127.0.0.1:58370)
0925 05:14:44.808852 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.808916 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.809637 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.809916 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.809918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.810176 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.810339 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.810343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.811010 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.811011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.812056 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.812059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.812662 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.812663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.812800 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.813489 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.813498 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.815016 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.817052 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.817061 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.817063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.817379 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.817633 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.817634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.817634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.817739 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.818017 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.818024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.818183 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.818300 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.818481 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":4,"thread_start_us":130,"threads_started":1}
I20260925 05:14:44.819133 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.809149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.3.6.188:33051)
0925 05:14:44.809320 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.809325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.809368 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:14:44.809638 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.809647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.809675 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.809770 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.810499 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.811564 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.812633 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.813081 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.813137 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.815744 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.815780 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.815781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.815788 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.817375 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.817733 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.817736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.817737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.817795 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.818018 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.818020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.818020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.818176 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.818349 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.818419 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.818544 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260925 05:14:44.819321 25894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.190@KRBTEST.COM
I20260925 05:14:44.826999 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.819667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.3.6.189:43597)
0925 05:14:44.819742 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.819744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.819777 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:14:44.819804 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.819806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.819817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.819879 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.820200 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.820742 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.821491 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.821708 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.821764 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.823991 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.823993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.823993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.824003 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.825763 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.826125 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.826126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.826127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.826200 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.826451 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.826451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.826452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.826574 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.826628 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.826812 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.826893 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:14:44.827018 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.819580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.189:43597 (local address 127.0.0.1:33012)
0925 05:14:44.819632 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.819645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.819685 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.819894 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.819895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.820071 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.820124 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.820126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.820830 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.820831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.821405 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.821408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.821779 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.821780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.821848 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.822325 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.822330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.823887 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.825855 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.825856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.825856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.826040 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.826293 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.826294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.826294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.826377 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.826640 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.826641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.826701 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.826772 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.826898 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:14:44.827438 25894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:14:44.827966 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.819999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.3.6.190:42549)
0925 05:14:44.820036 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.820038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.820045 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:14:44.820117 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.820117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.820125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.820192 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.820652 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.821176 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.822036 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.822349 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.822421 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.824775 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.824777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.824778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.824788 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.826314 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.826687 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.826688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.826688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.826746 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.826969 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.826970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.826970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.827118 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.827344 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.827816 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.827899 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.188@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.6.189@KRBTEST.COM
I20260925 05:14:44.834685 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.827732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47434)
0925 05:14:44.827845 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.827857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.827904 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.828024 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.828025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.828183 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.828228 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.828230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.829098 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.829099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.829690 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.829692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.830125 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.830126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.830183 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.830581 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.830586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.832130 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.833693 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.833694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.833695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.833864 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.834064 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.834065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.834065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.834118 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.834361 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.834361 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.834399 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.834463 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.834568 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 05:14:44.834694 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.827811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.3.6.188:33051)
0925 05:14:44.827849 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.827851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.827887 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:14:44.827958 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.827959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.827970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.828013 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.828320 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.829019 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.829764 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.830057 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.830108 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.832218 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.832219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.832219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.832226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.833593 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.833945 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.833946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.833946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.833998 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.834183 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.834184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.834184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.834297 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.834371 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.834490 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.834616 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":17}
I20260925 05:14:44.836280 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.828641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.3.6.189:43597)
0925 05:14:44.828678 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.828680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.828688 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:14:44.828735 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.828735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.828744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.828803 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.829351 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.829848 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.830699 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.830905 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.830951 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.833484 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.833485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.833485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.833493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.835040 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.835475 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.835476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.835476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.835537 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.835712 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.835713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.835713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.835892 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.836059 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.836100 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.836177 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:14:44.836606 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: Bootstrap starting.
I20260925 05:14:44.837718 25894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:44.837823 3098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:14:44.838053 25894 log.cc:824] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:44.838929 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: No bootstrap required, opened a new log
I20260925 05:14:44.839128 25718 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "737a3ae773a04da6b13ef837e998384d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58372
I20260925 05:14:44.841197 25894 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.841399 25894 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65c20d5d0f54a4d9ec59af6b64d11a2, State: Initialized, Role: FOLLOWER
I20260925 05:14:44.841490 25894 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.841847 25894 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:44.841799 25900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.841922 25900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:14:44.843873 25911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:44.843950 25911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:44.844286 25826 master_runner.cc:186] Error getting master registration for 127.3.6.190:42549: OK, instance_id { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" instance_seqno: 1790313284456207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:14:44.846439 25826 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:14:44.874176 25819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } attempt: 3
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.189@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:14:44.884763 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.874831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.3.6.188:33051)
0925 05:14:44.874953 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.874955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.874966 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:14:44.874992 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.874993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.875002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.875290 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.875840 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.876630 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.878295 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.878621 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.878693 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.881432 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.881434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.881434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.881443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.883334 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.883736 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.883737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.883738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.883829 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.884093 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.884094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.884094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.884305 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.884378 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.884565 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.884657 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:14:44.884763 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.874653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47444)
0925 05:14:44.874728 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.874740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.874785 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.875416 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.875418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.875648 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.875711 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.875713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.876734 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.876735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.878186 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.878191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.878708 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.878710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.878778 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.879403 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.879410 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.881300 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.883438 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.883439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.883440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.883646 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.883909 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.883910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.883911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.884008 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.884388 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.884390 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.884456 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.884530 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.884655 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 05:14:44.886617 25819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: Bootstrap starting.
I20260925 05:14:44.887534 25819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:44.887910 25819 log.cc:824] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:44.888917 25819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: No bootstrap required, opened a new log
I20260925 05:14:44.891088 25819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.891258 25819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 872f3ae001164b95ae84810b2be92793, State: Initialized, Role: FOLLOWER
I20260925 05:14:44.891353 25819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.891796 25819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:44.891702 25914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.892030 25914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:14:44.894011 25925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:14:44.894102 25925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:14:44.894189 25751 master_runner.cc:186] Error getting master registration for 127.3.6.190:42549: OK, instance_id { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" instance_seqno: 1790313284456207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:14:44.897322 25751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:14:44.900269 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } attempt: 3
Sep 25 05:14:44 dist-test-slave-1lvv krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790313284, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.6.190@KRBTEST.COM for kudu/127.3.6.188@KRBTEST.COM
I20260925 05:14:44.909355 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.900830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.3.6.188:33051)
0925 05:14:44.900883 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.900885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.900895 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:14:44.900928 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.900929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.900940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.901023 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.901433 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.902203 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.903278 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.903587 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.903677 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.906137 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.906139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.906140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.906149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.907994 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.908479 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.908480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.908481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.908548 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.908767 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.908767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.908768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.908921 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.908993 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.909168 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.909257 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:14:44.909360 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.900699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.6.188:33051 (local address 127.0.0.1:47446)
0925 05:14:44.900763 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 05:14:44.900776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:14:44.900822 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:14:44.901030 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:14:44.901031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:14:44.901272 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:14:44.901332 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.901335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.902316 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.902317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.903172 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:14:44.903175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.903678 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.903680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:14:44.903784 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.904331 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:14:44.904336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:14:44.906004 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:14:44.908136 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.908137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.908138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.908404 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.908627 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:14:44.908628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:14:44.908629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:14:44.908704 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:14:44.908989 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:14:44.908990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:14:44.909047 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 05:14:44.909126 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:14:44.909247 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:14:44.911854 25744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d: Bootstrap starting.
I20260925 05:14:44.912774 25744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d: Neither blocks nor log segments found. Creating new log.
I20260925 05:14:44.913094 25744 log.cc:824] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:14:44.914280 25744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d: No bootstrap required, opened a new log
I20260925 05:14:44.916570 25744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.916901 25744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737a3ae773a04da6b13ef837e998384d, State: Initialized, Role: FOLLOWER
I20260925 05:14:44.917093 25744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.917616 25928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.917728 25928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:44.917821 25718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:14:44.917891 25718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:44.918777 25718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.919027 25718 leader_election.cc:288] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [CANDIDATE]: Term 1 election: Requested vote from peers 872f3ae001164b95ae84810b2be92793 (127.3.6.189:43597), f65c20d5d0f54a4d9ec59af6b64d11a2 (127.3.6.188:33051)
I20260925 05:14:44.919502 25793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737a3ae773a04da6b13ef837e998384d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "872f3ae001164b95ae84810b2be92793"
I20260925 05:14:44.919642 25793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:44.919950 25868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737a3ae773a04da6b13ef837e998384d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2"
I20260925 05:14:44.920099 25868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:14:44.920644 25793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737a3ae773a04da6b13ef837e998384d in term 1.
I20260925 05:14:44.920984 25694 leader_election.cc:302] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737a3ae773a04da6b13ef837e998384d, 872f3ae001164b95ae84810b2be92793; no voters:
I20260925 05:14:44.921036 25868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737a3ae773a04da6b13ef837e998384d in term 1.
I20260925 05:14:44.921327 25744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:14:44.921830 25928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:14:44.922017 25928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [term 1 LEADER]: Becoming Leader. State: Replica: 737a3ae773a04da6b13ef837e998384d, State: Running, Role: LEADER
I20260925 05:14:44.922135 25928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } }
I20260925 05:14:44.922593 25928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737a3ae773a04da6b13ef837e998384d. Latest consensus state: current_term: 1 leader_uuid: "737a3ae773a04da6b13ef837e998384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.922679 25928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:44.922857 25941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:14:44.923434 25941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:14:44.928330 25868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [term 1 FOLLOWER]: Refusing update from remote peer 737a3ae773a04da6b13ef837e998384d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:44.928802 25793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [term 1 FOLLOWER]: Refusing update from remote peer 737a3ae773a04da6b13ef837e998384d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:14:44.928886 25928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:44.929168 25928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [LEADER]: Connected to new peer: Peer: permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:14:44.930197 25900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737a3ae773a04da6b13ef837e998384d. Latest consensus state: current_term: 1 leader_uuid: "737a3ae773a04da6b13ef837e998384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.930290 25900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:44.930451 25914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737a3ae773a04da6b13ef837e998384d. Latest consensus state: current_term: 1 leader_uuid: "737a3ae773a04da6b13ef837e998384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.930548 25914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:44.931599 25934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737a3ae773a04da6b13ef837e998384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.931682 25934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:44.931742 25948 mvcc.cc:205] Tried to move back new op lower bound from 7333123215059410944 to 7333123215042129920. Current Snapshot: MvccSnapshot[applied={T|T < 7333123215059410944}]
I20260925 05:14:44.932098 25914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737a3ae773a04da6b13ef837e998384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.932176 25914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:44.932447 25954 mvcc.cc:205] Tried to move back new op lower bound from 7333123215059410944 to 7333123215042129920. Current Snapshot: MvccSnapshot[applied={T|T < 7333123215059410944}]
I20260925 05:14:44.940855 25941 catalog_manager.cc:1347] Generated new cluster ID: cac1de8432a1403c97a7512ce9a2240b
I20260925 05:14:44.940935 25941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:14:44.946897 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.928039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.3.6.190:42549)
0925 05:14:44.928085 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.928087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.935942 (+ 7855us) server_negotiation.cc:407] Connection header received
0925 05:14:44.936055 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.936056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.936067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.936100 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.936975 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.937735 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.938904 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.939203 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.939247 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.942979 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.942981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.942981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.942991 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.944864 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.945287 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.945288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.945289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.945342 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.945516 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.945517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.945517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.945645 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.945713 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.946716 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.946797 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:14:44.951547 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.928555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.3.6.189:43597)
0925 05:14:44.928616 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.928619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.938597 (+ 9978us) server_negotiation.cc:407] Connection header received
0925 05:14:44.939563 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.939565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.939580 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.939612 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.941185 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.941942 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.945894 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.946205 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.946244 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.948761 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.948763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.948763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.948772 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.950361 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.950745 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.950746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.950746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.950802 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.950973 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.950973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.950974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.951097 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.951191 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.951370 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.951450 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:14:44.953718 25934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737a3ae773a04da6b13ef837e998384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.953836 25934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d [sys.catalog]: This master's current role is: LEADER
I20260925 05:14:44.954603 25952 mvcc.cc:205] Tried to move back new op lower bound from 7333123215059410944 to 7333123215042129920. Current Snapshot: MvccSnapshot[applied={T|T < 7333123215059410944}]
I20260925 05:14:44.954768 25900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737a3ae773a04da6b13ef837e998384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737a3ae773a04da6b13ef837e998384d" member_type: VOTER last_known_addr { host: "127.3.6.190" port: 42549 } } peers { permanent_uuid: "872f3ae001164b95ae84810b2be92793" member_type: VOTER last_known_addr { host: "127.3.6.189" port: 43597 } } peers { permanent_uuid: "f65c20d5d0f54a4d9ec59af6b64d11a2" member_type: VOTER last_known_addr { host: "127.3.6.188" port: 33051 } } }
I20260925 05:14:44.954878 25900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:14:44.956764 25941 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:14:44.957624 25941 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:14:44.961210 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:44.928723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.3.6.188:33051)
0925 05:14:44.928762 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:44.928764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:44.943810 (+ 15046us) server_negotiation.cc:407] Connection header received
0925 05:14:44.943845 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:44.943846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:44.943860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:44.943931 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:44.950288 (+ 6357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.951019 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.952139 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:44.952399 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:44.952439 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:44.953904 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:14:44.953905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:14:44.953905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:14:44.953914 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:14:44.956661 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.960285 (+ 3624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.960287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.960288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.960359 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:14:44.960559 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:14:44.960576 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:14:44.960577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:14:44.960716 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:14:44.960805 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:44.961013 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 05:14:44.961104 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:14:44.975862 25941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737a3ae773a04da6b13ef837e998384d: Generated new TSK 0
I20260925 05:14:44.976171 25941 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:14:45.003118 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.001481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.3.6.189:43597)
0925 05:14:45.001534 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.001536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.002237 (+ 701us) server_negotiation.cc:407] Connection header received
0925 05:14:45.002364 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.002366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.002378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.002412 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.003032 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33036: BlockingRecv error: recv got EOF from 127.0.0.1:33036 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 05:14:45.004585 25898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.001908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.3.6.188:33051)
0925 05:14:45.001951 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.001953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.003815 (+ 1862us) server_negotiation.cc:407] Connection header received
0925 05:14:45.004038 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.004039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.004052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.004085 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.004512 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47464: BlockingRecv error: recv got EOF from 127.0.0.1:47464 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 05:14:45.005277 25747 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.003444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.3.6.190:42549)
0925 05:14:45.003488 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.003490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.003923 (+ 433us) server_negotiation.cc:407] Connection header received
0925 05:14:45.004771 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.004773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.004787 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.004822 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.005206 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58388: BlockingRecv error: recv got EOF from 127.0.0.1:58388 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 05:14:45.007624 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.006712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.3.6.189:43597)
0925 05:14:45.006754 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.006756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.007006 (+ 250us) server_negotiation.cc:407] Connection header received
0925 05:14:45.007131 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.007133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.007155 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.007189 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.007566 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33040: BlockingRecv error: recv got EOF from 127.0.0.1:33040 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 05:14:45.010967 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.010041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.3.6.189:43597)
0925 05:14:45.010080 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.010082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.010349 (+ 267us) server_negotiation.cc:407] Connection header received
0925 05:14:45.010461 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.010462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.010473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.010507 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.010902 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33042: BlockingRecv error: recv got EOF from 127.0.0.1:33042 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 05:14:45.015300 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.014307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.3.6.189:43597)
0925 05:14:45.014353 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.014355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.014676 (+ 321us) server_negotiation.cc:407] Connection header received
0925 05:14:45.014793 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.014795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.014808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.014842 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.015237 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33056: BlockingRecv error: recv got EOF from 127.0.0.1:33056 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 05:14:45.021883 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.020839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.3.6.189:43597)
0925 05:14:45.020889 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.020892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.021199 (+ 307us) server_negotiation.cc:407] Connection header received
0925 05:14:45.021316 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.021317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.021330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.021365 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.021814 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33072: BlockingRecv error: recv got EOF from 127.0.0.1:33072 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 05:14:45.032409 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.031309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.3.6.189:43597)
0925 05:14:45.031360 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.031362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.031684 (+ 322us) server_negotiation.cc:407] Connection header received
0925 05:14:45.031851 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.031852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.031866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.031900 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.032329 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33074: BlockingRecv error: recv got EOF from 127.0.0.1:33074 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 05:14:45.050911 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.049945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.3.6.189:43597)
0925 05:14:45.049999 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.050002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.050142 (+ 140us) server_negotiation.cc:407] Connection header received
0925 05:14:45.050208 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.050210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.050224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.050317 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.050843 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33082: BlockingRecv error: recv got EOF from 127.0.0.1:33082 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 05:14:45.084930 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.084075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.3.6.189:43597)
0925 05:14:45.084136 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.084138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.084223 (+ 85us) server_negotiation.cc:407] Connection header received
0925 05:14:45.084308 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.084309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.084319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.084373 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.084848 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33090: BlockingRecv error: recv got EOF from 127.0.0.1:33090 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 05:14:45.150981 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.150219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.3.6.189:43597)
0925 05:14:45.150298 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.150300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.150326 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:14:45.150402 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.150403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.150414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.150469 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.150904 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33106: BlockingRecv error: recv got EOF from 127.0.0.1:33106 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 05:14:45.281065 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.280261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.3.6.189:43597)
0925 05:14:45.280358 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.280361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.280375 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:14:45.280455 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.280456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.280467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.280524 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.280990 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33120: BlockingRecv error: recv got EOF from 127.0.0.1:33120 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 05:14:45.539095 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:14:45.538386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.3.6.189:43597)
0925 05:14:45.538457 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:45.538460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:45.538471 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:14:45.538538 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:45.538539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:45.538550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:45.538616 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:14:45.538989 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33136: BlockingRecv error: recv got EOF from 127.0.0.1:33136 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:14:45.844439 25911 catalog_manager.cc:1259] Loaded cluster ID: cac1de8432a1403c97a7512ce9a2240b
I20260925 05:14:45.844527 25911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: loading cluster ID for follower catalog manager: success
I20260925 05:14:45.845299 25911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: acquiring CA information for follower catalog manager: success
I20260925 05:14:45.845556 25911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f65c20d5d0f54a4d9ec59af6b64d11a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:14:45.894644 25925 catalog_manager.cc:1259] Loaded cluster ID: cac1de8432a1403c97a7512ce9a2240b
I20260925 05:14:45.894734 25925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: loading cluster ID for follower catalog manager: success
I20260925 05:14:45.895535 25925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: acquiring CA information for follower catalog manager: success
I20260925 05:14:45.895831 25925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 872f3ae001164b95ae84810b2be92793: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:14:46.055088 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:46.052424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.3.6.189:43597)
0925 05:14:46.052547 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:46.052549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:46.052561 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:46.052624 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:46.052625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:46.052682 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:46.052755 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:46.052901 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.053404 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.054170 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.054397 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.054469 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:46.054661 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:46.054808 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:46.054889 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:46.054988 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 05:14:46.055014 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260925 05:14:46.059432 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:46.057048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.3.6.188:33051)
0925 05:14:46.057174 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:46.057176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:46.057209 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:14:46.057246 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:46.057247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:46.057285 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:46.057352 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:46.057456 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.057942 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.058656 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.058892 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.058948 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:46.059079 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:46.059186 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:46.059251 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:46.059297 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 05:14:46.059315 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260925 05:14:46.064309 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:46.061493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.3.6.188:33051)
0925 05:14:46.061540 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:46.061543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:46.061646 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:14:46.061739 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:46.061741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:46.061752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:46.061801 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:46.062000 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.062711 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.063524 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.063790 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.063855 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:46.064022 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:46.064060 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:46.064090 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:46.064204 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 05:14:46.064232 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 05:14:46.064543 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:46.061237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.3.6.189:43597)
0925 05:14:46.061493 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:46.061495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:46.061648 (+ 153us) server_negotiation.cc:407] Connection header received
0925 05:14:46.061687 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:46.061688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:46.061701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:46.061756 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:46.061921 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.062698 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.063678 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.064012 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.064079 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:46.064235 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:46.064283 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:46.064318 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:46.064435 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 05:14:46.064469 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260925 05:14:46.064754 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:14:46.061217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.3.6.190:42549)
0925 05:14:46.061409 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:14:46.061412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:14:46.061424 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:14:46.061476 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:14:46.061477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:14:46.061541 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:14:46.061789 (+ 248us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:14:46.061834 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.062408 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.063534 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:14:46.063906 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:14:46.063980 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:14:46.064218 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:14:46.064351 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:14:46.064600 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 05:14:46.064620 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 05:14:46.064648 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260925 05:14:46.066577 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25676
I20260925 05:14:46.072486 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25751
I20260925 05:14:46.078063 3098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH05q1Z/build/release/bin/kudu with pid 25826
2026-09-25T05:14:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2955 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5838 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277713 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST